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The main purpose of the experiment was to answer the 
question wnether bread or narrow terms function most effectively in 
the retrieval ot relevant documents. The answer depends on wuat the 
user ot the retrieval system wants, his wants being expressed in 
terms of stated prec ision-recaxl preference# or by the exact number 
of relevant documents he wishes to retrieve. Depth of indexing does 
not contribute significantly to effective retrieval. Documents should 
be indexed with broad terns to satisfy recall preferences with narrow 
terms to satisfy precision preferences since the amount of material 
retrieved in a system is not a simple function of the total numbers 
of terms posted to documents in the collection. At high cut-off 
values the retrieval power of broad and narrow terns tends to become 
equalized. Precision can be improved through raising the cut-off 
point and deleting broad terms. Experiments with weighted indexing 
provided poor results, and with title-term indexing gave inconclusive 
results. (AB) 
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CHAPTER I 



THE MEANING OF SPECIFICITY 
Introduction 

The study undertaken here is in the area of indexing theory. 
The question to which the study is addressed is how specific 
should index terms be. So worded, the question poses difficul- 
ties. It is not clear, for instance, what is meant by ”specifi- 
city.** **How specific is specific?** is a question raised in the 
library literature whenever the **principle of specific entry** 
comes under discussion. 

The specific entry principle is an injunction to catalogers to 
deal with language in a certain way -- to find the most specific 
subject heading for a work being indexed. It assiimes that one can 
say in a reasonably simple way what a book or document is about — 
specifically, where specifically is illustrated by reference to 
inclusive classes: 

Enter a work under its subject-heading, not 
under the heading of a class which includes 
that subject. Ex. Put Lady Cust*s book on 
**The Cat** under CAT, not under ZOOLOGY or 
MAMMALS or DOMESTIC ANIMALS; ^ 

The principle of specific entry is usually acknowledged as a 

^Charles A. Cutter. Rules for a Printed Dictionary Catalog 
(Washington, D.C.: U.S. Government Printing Office, 1876), p. 37. 



1 



2 



tenet inviolable in the practice of assigning subject headings. 
Indeed it would seem that no other principle or rule of indexing^ 
enjoys a similar stature. Yet it has been observed that the rela- 
tivity of the principle makes it difficult to apply or comprehend. 

A problem then in the study under consideration is to find a way of 
coping with the apparent elusiveness of the concept of specificity. 
In particular what is needed is a definition of "specificity" which 
is clear and distinct, suitable for use as a variable in a theoret- 
ical or experiment study of indexing effectiveness. In order to 
give something more than a speculative discussion of specificity, 
a procedure must be found for objectively evaluating indexing at 
varying levels of specificity. Such a procedure must Include a 
method for systematically varying the specificity of indexing. Also 
required is a method for measuring which of the "more or less" spe- 
cific indexings is the most effective in retrieval performance, in 
the sense of enabling users to find all and only those documents 
relevant to their needs. These are among the matters discussed in 
the following pages which propose an approach toward the problem of 
evaluating specificity in indexing. 

There are four chapters . Chapter I discusses the background 
of the problem: the first appearance of a need for specific index- 

ing in the nineteenth century and the consequent concern for. speci- 
ficity reflected in the development of subject and title catalogs, 
the continual confusion over the meaning of specificity, in particu- 
lar the relativity of its meaning. Seven definitions of specificity 




The viewpoint is taken that in the assigning of terms to books 
or documents there is no essential difference between subject cata- 
loging, information indexing and classifying. 
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are considered. One of these, called "operational specificity," 
is selected for use in an experimental study of indexing effective- 



ness. This definition is adequate in the sense that it gives a ^ 
quantitative measure of the specificity of an indexed collection o^ 
documents and provides a method whereby the specificity of index- 
ing can be varied in a systematic way. It is argued that this def- 
inition also reasonably explicates what is vaguely understood by 
specificity" in the minds of those following the specific entry 
principle when assigning subject headings. 

The core of the present study is an experiment designed to 
test whether specific indexing is better than non-specific or broac 



indexing. The experiment is a retrieval experiment; that is, doc- 
uments are retrieved from an indexed collection of documents in 
response to search questions addressed to the collection. The out- 
put is then evaluated according to one of the usual measures of re 
trieval effectiveness. In the experiment the indexing of the docu- 
ment collection is altered by deleting sets of terms, broad terms 
and narrow terms. After each alteration in indexing, retrieval and 
evaluation procedures are carried out. Obtained thus are different 
measures of retrieval performance for indexings of varying specifi- 
cities. The data used in the experiment is the Salton-Cranfield 
data, consisting of 200 documents in the field of aeronautics , 42 
search questions addressable to the document collection and relevanice 

I 

judgments for each document with respect to each of the questions, j 

i 

Chapter II outlines the experimental design. Results are analyzed | 
in Chapter III, Chapter IV summarizes the results and speculates ■ 
on their practical applicability both within the constraints of | 
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traditional library operations and with regard to what may be 
possible in libraries of the future. 



Background 



The problem of specificity made its appearance in the library 

literature in the middle of the nineteenth century. At this time 

» 

there was indication of a growing dissatisfaction with the classed 
or classified catalog. Quite apart from the view which regarded the 
classification of books as a "logical absurdity,"^ retrieval diffi- 
culties were encountered on a practical level. Samsom Low, for in- 
stance, complained that 

under the old system of classification (the classi- ’ 
fied catalog) the difficulty has always been to 
find a given title, though enabled to find a group 2 
of books published within a scientific discipline. 

As a response to the difficulties it was proposed that books should 
le indexed according to the words in their titles. Crestadoro was 
the foremost proponent of title-term indexing. Proposing a concord- 
ance of titles which "follows out each author's own definition of 

3 

his book," he put his principles into practice first in the 



Jevon's famous remark. For an account of the theoretical 
opposition to classified catalogs see J. Metcalfe, Subject Classi- 
fying and Indexing of Libraries and Literature (New York: Scare- 

crow Press, 1959), p. 32 ff. 

2 

Quoted in J, Metcalfe, Alphabetical Subject Indication of 
Information . Vol. Ill of Rutgers Series on Systems for the Intel- 
lectual Organization of Information, ed. Susan Artandi (New 
Brunswick, N.J.: Rutgers -- The State University, Graduate School 

of Library Service, 1965), p. 25. 

o 

•^From his Manchester Catalogues. Quoted in J. Metcalfe, 
Information Indexing (New York: Scarecrow Press, 1957), p. 48. 
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British Catalog and then in the catalogs he prepared for the 

Manchester Public Free Libraries. Poole also advocated title-term 

indexing -- but with reservations: 

In most cases the author’s own title best ex- 
presses the subject -- but if he has given it 
an obsolete or fanciful title, the indexer will 
give it a better one, and will place it under 
the heading where it naturally belongs.^ 

The import of the reference is clear* It is necessary that if 

some sort of rapport is to be established between the catalog and 

the catalog user, the catalog must use a language relatively free 

of surprises, one that is at least somewhat predictable. Crestadoro 

himself was quite aware of the weakness of title-term indexing and 

he warned the users of his catalog that some relevant items would 

be missed: "under any given subject the whole of the books . . . 

are not brought together, but only those in which the name of the 

2 

subject occurs in the title." The arguments for a uniform lan^ 
guage and for completeness led to the development of the alphabet- 
ical subject catalog -- a catalog of standard subject headings al- 
phabetically arranged. While dealing a "death blow" to title- 
term indexing was a purpose in the creation of the alphabetical sub- 
ject catalog, i^t was, however, subordinate to the more important 
purpose of providing a viable and constructive alternative to the 
classified catalog. The classified catalog, by permitting generic 



^Quoted in Metcalfe, Alphabetical Subject Indication of In - 
formation , p, 27. 

2 

Quoted in Metcalfe, Information Indexing ^ p. 48. 

o 

Metcalfe in Information Indexing quotes Taylor as telling 
British students that "as far as British and American cataloguers 
are concerned, its (title-term entry) death blow was dealt by 
Cutter," p. 47. 
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entry :nade it difficult to find books on a specific subject. 

It was this lack of specificity that the alphabetical subject cata- 
log intended to remedy. The Principle of Specific Entry was 
stated by Cutter in 1876: "Enter a work under its subject heading, 

not ij^nder the heading of a class which includes that subject."^ 

And in 1951 the raison d^etre of the subject catalog seems virtually 
unchanged: "The primary purpose of the subject catalog is to show 

which books on a specific subject the library possesses." Some- 
what regrettably perhaps, for it obscured the issues, the opposition 
of specific w. generic entry continued to be regarded as part of 
the controversy over the relative merits of the two kinds of cata- 
logs. The controversy goes on still today, though with diminishing 
force, especially in America where the alphabetical subject catalog 
is well entrenched in most libraries and the principle of specific 
entry is, in practice at least, everywhere accepted. 

The Problem; Relativity of Specificity 

Confusion has centered around the specific entry principle 
from the time it was first formulated at the end of the last cen- 
tury. A recent expression of the confusion is Dunkin' s complaint, 
made in the summer of 1967: 

'Specificity' is a magic word which we all accept 
but seldom really define. 'Specificity' means 
all things to all men and little to most, partic- 
ularly when we are prepared to abandon our 

^Cutter, Rules for a Printed Dictionary Catalog , p. 37. 

^ David J. Haykin, Subject Headings: A Practical Guide 

(Washington, D.C.: U.S. Government Printing Office, 1951), p. 1. 

lo 
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definition at any point where the 'convenience 
of the public’ can be argued.^ 

The question ”how specific is specific?” is a recurrent one in the 
literature of library science. Often it is simply asked rhetori- 
cally for the purpose of showing that there exists a problem of 
viewpoint; 



How specific is specific entry? As shown, 

Cutter was specific when he used Shetland wool 
and Merino sheep. Framework knitters, Handloom 
weaver, and Pneumatic loom. But . . • Cutter 
was living in simpler times. Would he have been 
as specific as a model of a make of motor car for 
the model's manual; for example, the manual of 
the Morris Mini Motor Car? 

Perhaps the most direct assault on the relativity of "specificity” 
was that made by O.I. Lilley in 1955. He answered the question 
"How specific is ’specific'?” with "Well, it all depends!"^ 

Constructive answers to the question generally refer to a 
"public.” Cutter, for instance, demanded that the specific entry 
principle be viewed in light of another principle, viz , the usage 
principle which requires that headings be chosen from the language 
of a particular public, the public for whom the catalog is intend- 
ed, rather than from the indexer's own language or even that of 
the document. Metcalfe argued that, in practice, subject headings 
are not always as specific as the subject of the work and that one 
of the reasons for this is that the catalog user must get what he 



^Paul S. Dunkin, "Cataloging and CCS: 1957-1966,” Library 

Resources and Technical Services . II (Summer 1967), 284. 

2 

J, Metcalfe, Alphabetical Subject Indication of Infoinnation , 

p. 35. 

^Oliver L. Lilley, "How Specific is 'Specific'? ” Journal of 
Cataloging and Classification ^ II (January 1955), 8. 
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wants by a name which he knows, without any guesswork on his part.^ 

It is rather the indexer's obligation to guess at the degree of sub- 
ject specification which will best serve his public. The public is 
the operative factor. 

It is not immediately clear what good can be accomplished by 
introducing a public. It is doubtful, for instance, that a cata- 
loger instructed to index a document as specifically as possible 
from the point of view of the public served is likely to have ap- 
preciably better insight into what he is doing than were he told 
simply to index with specific terms. Findings of a consensus set 
study conducted as part of an Indexing Project at the Graduate Li- 
brary School suggest that while indexers agree on the important con- 
cepts to index, they alter these in various ways when verbalizing 

them into index terms, and that one of the most usual alterations is 

2 

changing the level of specificity. And again, in another study on 
the Indexing Project, it was found that disagreement about the 
choice of index terms is, in large part, attributable to a difference 
of opinion as to the appropriate length of phrase to be taken as an 
index term, again suggesting that much of the inconsistency among 
indexers is due to disagreement about degree of subject specifica- 

3 

tion. If on the appropriate degree of subject specification, coin- 
ed. Metcalfe, Subject Classifying (New York: Scarecrow Press, 

1959), p. 51. 

2 

W. Boyd Rayward and Elaine Svenonius, "Consistency, Consensus 
Sets and Random Deletion," Studies in Indexing Depth and Retrieval 
Effectiveness (NSF GN 380) (Chicago: University of Chicago, 

Graduate Library School, 1967), p. 3. 
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cidence of point of view is rare among indexers describing the 
same document — one wonders how much rarer still it must be be- 
tween an indexer and a public who might find the document useful? 
After all^ a public is single only when regarded as a vague undif- 
ferentiated mass; scrutinized more closely it appears fragmented in- 
to as many different points of view as there are individuals making 
up the public. Seen this way, one may agree with Lilley that "spec- 
ificity may be so intangible as to be nothing more in an effective 
sense, than a chance relationship between the user's need of the 
moment and the format of a particular book an individual library 
happens to own."^ Coming back to the question: how specific is 

specific? -- invoking a "public'* would seem not to provide hope of 
a useful answer. It shows specificity to be entirely viewpoint 
dependent. 

Questions of viewpoint are very like questions of relevance; 
both attest to the rights of individual imagination, and both work 
against the exploitation of language for the purpose of retrieving 
information. The position outlined above, which is ready to reject 
the concept of specificity because it is so much a matter of view- 
point, has a semantic parallel in a definite and rather extreme at- 
titude towards language in general. According to this view the 
meaning of a word is a variable; it varies according to context. 

Some meanings are more stable than other, for instance those of 
technical terms, but even these are subject to flux, shaped by the 
meanings of contiguous words on the written page as well as by the 

^Lilley, "How Specific Is 'Specific'? " p. 6. 
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net of associations elicited in the mind of an individual reader. 

Meaning is itself viewpoint dependent; so much so that it has been 

argued that no two uses of a word are quite the same, each use is 

a special case of meaning, there being no single meaning which is 

1 

fixed and inviolate. 

The problem with specificity is in fact a problem in semantics, 
since specificity is a property of meaning. One word differs in 
specificity from another -- it is a rough way of saying that there 
is a difference in specificity of meaning. It is hardly surprising 
that specificity of meaning should be viewpoint dependent, when 
meaning itself is. Both vary according to context and individual 
mind-sets or dispositions towards the use of language. Lilley gives 
an example where the word Botany is specific in the context of de- 
partmental budgets, but general in the context of a group of bota- 
nists : 



Within the subject area of departmental budgets 
. . • the words "Botany'* and **Geology" reason- 
ably can be called specific terms . . . But in a 
group of botanists, if one individual were to re- 
mark that his special interest is "Botany**, the 
statement would be so imprecise as to lack 
meaning for his hearers. In this instance **Bo- 
tany*’ is no longer a specific term, but has 
become a very general one.^ 

Focusing on the relativity of specificity then may be regarded 
as a particular expression of a more general view of language. 
Carrying this view to an extreme, to a reductio ad absurdum, one 
can develop an argument for silence. Frege suggests that a word 



^Ludwig Wittgenstein, Philosophical Investigations , trans. 
G.E.M. Anscombe (New. York: Macmillan, 1953), see esp. p. 66. 

^Lilley, **How Specific Is 'Specific*? *' p. 4. 
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with vague boundaries cannot be called a word at all*^ Rejecting 
so much rather commits one to semantic solipsism • « • as perhaps 
skepticism is the "price” of a too intense demand for certain knowl 
edge. 

*But is a blurred concept a concept at all?*-- 
Is an indistinct photograph a picture of a 
person at all? Is it even always an advan- 
tage to replace an indistinct picture by a 
sharp one? Isn't the indistinct one often 
exactly what we need?^ 

Wittgenstein, Philosophical 
Investigations 

The main argument against semantic solipsism is that confusion 
and silence do not in fact represent what is generally the case* 
Language works in a dynamic way and words to be flexible enough for 
every occasion of their use must be somewhat vague and ambiguous. 
While this contributes to making language at some very basic level 
essentially private, it does not make communication impossible* A 
point to be noted in this connection is that it is possible for a 
word to be ambiguous in a very clear way. For instance pointer 
words such as "this" and "that" are extremely ambiguous, techni- 
cally speaking, in that their possible referents are so extremely 
multiple; yet these words are generally used in such a way that 
their meaning is clear. There are really many situations where a 
"relative" word is needed. The meanings of words are stable as 
well as variable; it is a matter of degree* Meanings acquire a 
certain fixity inasmuch as they are used in accordance with lin- 

^See quoted in Wittgenstein, Philosophical Investigations , 
p. 34. 

2 

Wittgenstein, Philosophical Investigations * p. 34. 
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guistic habits or conventions* Wittgenstein likens these conven- 
tions and habits to rules of a game. There are really many lan- 
guage games, and their rules, more or less acknowledged, have the 
effect of forcing a degree of standardization more or less suffi- 
cient for communication and understanding. The notion of consensus 
is important here -- appealing to a consensus in the use of words to 
justify the possibility of communication, and in particular the re- 
trieval of information* It is possible then, without denying the 
variability and conflict of viewpoints, to ask what is the nature 
and extent of the overlapping area of agreement about the specifi- 
city of index terms. Proof that there is some area of agreement is 
given in the Consensus Set Study mentioned on page 8.^ 

Another argument for not rejecting *’specif icity" on the grounds 
that it means different things to different people is that it may 
very well be a useful concept. Wittgenstein; "I use the name *N* 
(substitute ’specificity*) without a fixed meaning. (But that de- 
tracts as little from its usefulness, as it detracts from that of 
a table that it stands on three legs instead of four and so some- 
times wobbles.") This suggests an answer to the question: why, 

if no one seems to know what is meant by "specific," is the princi- 
ple of specific entry so generally accepted? The answer here is a 
pragmatic one. Librarians have found the concept of specificity, 
as it is embodied in the specific entry principle, a useful one. 

It has served some purpose; 

^Rayward and Svenonius, "Consistency, Consensus Sets and Ran- 
dom Deletion." 

^Wittgenstein, Philosophical Investigations , p. 37. 
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We librarians take no little pride in the fact 
that entries in our subject card catalogs are 
made under ''specific’* headings. This doctrine 
of specificity we tell ourselves (and others), 
not only simplifies our own task of putting in- 
formation into the catalog, but in turn sim- 
plifies the user's job of getting the informa- 
tion out again; ^ 

Lilley goes on to suggest that librarians in their pride are guilty 
of self-deception since the word "specific" has no fixed meaning. 
The answer to that is: nevertheless, the meaning may be as precise 

as it needs to be. The fact that the doctrine of specificity is 
so widely accepted strongly suggests th£ t it "works*" The argu- 
ment of pragmatism is, like the argument of consensus, a rather 
standard ploy in staving off the skepticism entailed in a position 
of epistemological relativity. With "specificity" it is a question 
of linguistic relativity. 




The Uses of Specificity 

The worry about specificity in library literature arises in 
part from the fact that the specific entry principle seems to be 
regarded as useful. Arguments for the usefulness of the principle 
are principally of three kinds. The first holds that subject head- 
ings should be specific because the demand of users is for specific 
and detailed information. The second is that it serves as a deci- 
sion-making device -- if the most specific heading is not assigned, 
then where among the lattice-like branchings of possible subject 
headings is the "best" one to be found? Thirdly, choosing specific 
subject headings is a way of insuring that too many entries do not 

^Lilley, "How Specific Is 'Specific'?," p. 3. 
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accumulate under any one heading, a situation to be avoided be- 
cause it would make searching tedious. These arguments are consid- 
ered below. 

Julia Pettee writes that the increasing specialization of 
civilization has created the need for specific and detailed infor- 
mation: 



it (our present dictionary catalog) arose in 
response to a demand of our highly specialized 
civilization for specific and detailed infor- 
mationi To gain this information from trea- 
tises or through some classified system is more 
laborious and time consuming than by means of 
alphabetical captions to which one can go di- 
rectly without an intermediary symbol. Hence 
Cutter's primary rule for the dictionary cata- 
log 'Enter under specific topic’, ^ 

The same argument, that the public wants specific as opposed to 

"classified” information, is given by Metcalfe: 

This reaches down to the basic reason for 
Cutter’s alphabetico-specific entry; the 
catalogue user must get to what he wants by 
a name of his subject which he knows, in an 
order which he knows, without any guesswork 
classification on his part. 2 

Whether or not the demand for specific information is real or by 
intuition only assumed, is still, as it has been for more than 
half a century, a matter of speculation and opinion. It is true 
there have been many statistical studies of users' needs, but 
these tend to be unsatisfactory because of defects in methodology 
and in the analysis of results and because of questionable start- 



^ Julia Pettee, Subject Headings: The History and Theory of 

the Alphabetical Subject Approach to Books (New York: H,W, Wilson, 

1946), p. 57, 

2 

Metcalfe, Subject Classifying , p, 51, 
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ing assumptions.^ Most of these studies seem aimed at trying to 

find out whether the subject catalog is a useful device for finding 

information. The subject catalog, since it was developed in opposi- 

tion to the classified catalog and is based on the principle of 

specific entry, should, if Pette.e’s reasoning is valid, be useful 

to seekers of special and detailed information. The results of the 

studies, however, though admittedly tenuous, seem to indicate that 

no one would be greatly inconvenienced if the subject catalog were 

abandoned, at least in academic libraries: reference librarians 

rely mostly on reference works; research workers turn to subject 

3 

bibliographies; and scientists have methods of their own. Taking 
seriously these findings, one is led to wonder if perhaps this prod- 
uct, the subject catalog, that librarians are offering is not what 
it might be. In asking what might be wrong, the thought occurs 
that perhaps the principle of specific entry itself has in some way 
contributed to the catalog's failure to be relevant. By precluding 
broad subject headings when a specific one can be found, the prin- 
ciple has the effect of limiting access to the catalog. Broad terms 
may be useful. While a broad heading may fail to give a satisfactory 
description of a document, such a heading might, by its ubiquity, 
be reinforced in the memory of a user, and consequently be just the 

^Carlyle J. Frarey, Subject Headings > Vol. 1, Part 2 of The 
State of the Library Art, ed. Ralph R. Shaw (New Brunswick, N. J, : 
Rutgers -- The State University, Graduate School of Library Serv- 
ice, 1960) , p. 54. 

2 

See pp. 4-6. 

^Frarey, Subject Headings , p. 54. 
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one he would choose to look under. A word often seen can be read- 
ily remembered, there being no need for guess work or imagination. 
Further, the more specific the headings the less chance there is of 
the user finding all the material he needs under a given heading. 

For instance, a library may have no or a too brief account of the 
history of Virginia. However, a good deal on the history of 
Virginia will be found in histories of settlements in Eastern States. 
Thus even when the demand is for specific and detailed information, 
it could follow that this demand is more often satisfied in a system 
which can pexinit the user to broaden a search in a meaningful way 
when he is not immediately satisfied by what the system yields under 
a specii.'Lc heading. Pettee's argument seems weak. For subject 
cataloging and indexing in general, the question of whether broad 
or narrow terms are preferable seems independent of the "steadily 
increased demand for specific and detailed information." 

The second argument for the need for a principle of specific 
entry is based on the usefulness of the principle in making deci- 
sions -- if not the most specific heading, what then. The possibil- 
ities are too many. Haykin uses this argument: 

It is necessary, however, to state the reason for 
the use of the most specific heading applicable, 
rather than the broader heading which comprehends it. 

If the subject catalog were to consist of a 
predetermined number of more or less broad headings, 
a work on a specific topic would have to be entered 
under the broader one. The broader heading would 
thus be used for works as comprehensive as the 
heading, as well as for works on all the topics 
comprehended by it. To find out whether the li- 
brary possesses a book on a specific topic, the 
reader would, in the first place, need to know how 
broad a heading might be used for it, and, in the 
second place, would have to scan all the entries un-^ 
der the broader heading in order to select those 
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which are of interest to him • • * a reader 
looking for material on the income tax can be 
sure of finding it quickly and surely only by 
looking under the heading Income Tax # If the 
library were to choose a broader heading, it 
would have to be Taxation or even Finance, Pub - 
lic # Obviously, the number of entries under 
either would be relatively large in any library. 

The reader could not, in any case, be certain 
that he had hit upon the heading which the li- 
brary has, more or less arbitrarily, chosen for 
topics related to and including the income tax . . 

This argument -- and Pettee*s as well -- seems to be backed by the 
belief that a certain economy must be observed in the assigning 
of subject headings, viz # , that the choice between a broad or spe- 
cific heading must be an either-or-but-not-both option. The either 
or assumption, however, cannot be immediately dismissed. For one 
thing, belief in it has been strong enough to energize the century- 
long controversy over the relative merits of the classified vs . 
the alphabetic subject catalog, the classified catalog providing 
for ''generic'* entry. Though from a logical point of view it is a 
misleading use of the concept of class, in the language of library 
literature "classified" is often regarded as "nonspecific." Cutter 
for instance, in stating the specific entry principle uses "class" 
by way of a negative definition: "Enter a work under its subject- 

heading, not under the heading of a class which includes that sub- 
2 

ject." The dichotomy is partly physical in that the arrangement 
of a catalog can be classified or alphabetical but not both. But 
partly also it is the dichotomy of whether a user wishes to find 
under a heading all or most documents relevant to his needs, at the 
expense of going through many irrelevant documents, or whether he 



O 



^Haykin, Subject Headings; A Practical Guide , pp. 9 and 10, 
^Cutter, Rules for a Printed Dictionary Catalog , p. 37. 
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would prefer to avoid examining a large number of irrelevant doc- 
uments, but at the price of missing many of the possible relevant 
ones. This is the problem of whether it is better to maximize 
recall, the percentage of relevant documents retrieved, or to 
maximize precision, the percentage of retrieved documents which 
are relevant. A “tradeoff** seems to hold, for the most part: any 

improvement in recall results in a loss in precision, and, con- 
versely, that a bettering of precision makes for poorer recall. 

The choice between all or only relevant material is not unrelated 
to the choice between broad or specific headings. Samsom Low*s 
argument against the classified catalog, mentioned on page 4, seems 
to be an argument against access through broad terms and their 
potentiality for eliciting irrelevant material. The de facto as 
well as the theoretical history of library catalogs might be viewed 
as an attempt to reach some sort of acceptable compromise between 
broad and specific terms. Metcalfe points to **the steadily in- 
creased use of specific entry through the 19th and 20th centuries 
in the face of continuing propaganda for classified entry'*^ — which 
perhaps suggests not compromise really, but a tendency to favor 
specific terms over broad ones, good precision over good recall. 

On the other hand, the generic-specific controversy and the 
either-or assumption on which it is based could be misleading. It 
is not clear why access through broad or specific terms should be 
thought to exclude each other. It seems natural to ask why not have 
both: why not let the indexer choose all possible terms which seem 

^Metcalfe, Alphabetical Subject Indications of Information , 
p. 16. 




applicable, broad as well as specific, and let; the u.ser have the 
option of choosing the generic level at which he wishes to enter 
the system. The user would be well-served. He could, for instance 
himself choose precision at the expense of recall, or vice versa , 
depending upon the specificity or size of his requirement. This 
is the question of "double entry," and in the literature there are 
arguments pro and con. On the pro side Pettee points out that 
Cutter's rule for "specific entry" does not forbid the practice of 
double entry. He says 'that if room can be spared, the cataloger 
may put what he pleases under an extensive subject (a class), pro- 
vided he puts the less comprehensive works also under their re- 
spective specific headings.'^ And then Pettee goes on to observe 
that this practice of double entry "arose in respons€'. to the demand 
for ready reference which is the major function of the small and 
medium sized public library." "There is no doubt," she adds, "that 

these libraries have been benefitted far more than they have been 

2 

inconvenienced by this practice." As reasons for not using double 

3 

entry are cited the "cataloger's fallible judgment" and the fact 
that the "inconsistencies of these duplicate entries have caused 
much confusion."^ The inconsistencies and confusion presumably 
arise from the element of imagination which is introduced when cata 
logers are allowed to use subject headings more generic than the 

^Pettee, Subject Headings: The History and Theory of the 

Alphabetical Subject Approach to Books , p. 69. 
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subject of the work being cataloged; the cataloger's imagination 
may lead to the user's bewilderment. But this argument is not 
very convincing if, as Cutter advises, broad headings are used only 
to supplement and not to replace the specific ones. Another, and 
seemingly stronger, argument against double entry is that the li- 
cense it permits would burgeon the catalog with an unwieldy '*refer- 
ence apparatus,** disproportionate to the number of actual items 
processed. The danger of filling up the catalog is seen to be es- 
pecially serious in the case of large research libraries. Pettee 
writes : 

Certainly these duplicate entries should be 
carefully considered and watched, but it is 
only when the collections assume vast pro- 
portions and take on the major function of 
research that measures of more control and 
elimination become imperative. 

There is then some fear of losing control as the catalog increases 
in size and complexity. One might speculate that the fear need not 
necessarily be met by inhibiting the growth of the catalog. The 
possibilities offered by the computer might be seen as providing 
an alternative, in that the computer, as an organizing device, could 
make wieldy an elaborate reference apparatus far better than a phys- 

3 

ical collection of cards. In particular there would be no need to 



^Frarey, Subject Headings , p. 59. 

2 

Pettee, Subject Headings; The History and Theory of the Alpha 
betical Subject Approach to Books , p. 71. 

3 

The statement needs qualification. Unconstrained by the con- 
ventional image of a card catalog, the mind can imagine an enormous 
reference apparatus structured in physical space in a way that would 
permit a staff of thousands to perform searches. Manual access, 
theoretically at least, need not be clumsy or tedious. Ultimately 
the choice between a computerized system and a manual one is a mat- 
ter of cost. 
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choose either a classified or an alphabetical access method, since 
both would be possible. One entry only would be needed for each 
bibliographic item, and as many generic and specific added subject 
entries as needed could be provided in the form of an index. More- 
over, were the new media to prove economically feasible as well, 
the rationalization that users demand specific and detailed infor- 
mation would become superfluous and also some users' studies -- 
since users could demand anything they wanted and be satisfied. 

The third reason given for the practice of specific entry is 
that it serves to break up a collection of books or documents into 
clumps of a reasonable size. This is perhaps the most important 
reason -- at least it has been impossible to discuss the other two 
reasons without sensing the existence of this one in the background. 
The reasoning is that there must be upper and lower limits to the 
number of books posted to any given heading. For instance, a user 
would not be happy consulting a heading such as History--U,S. if 
all books in the library on this subject, no matter hoir incidently 
or specifically "related," were collected under this heading. The 
heading does not have sufficient discriminating power and thus im- 
poses on the would-be user the tedium of too much retrieval. That 
libraries are sensitive to the problem of over-retrieval is evidenced 
by the fact that where a very broad heading cannot be subdivided nat- 
urally, other devices, such as subdivision by publication date, have 

1 

been suggested to control the number of postings to the heading. 

^See Sidney L. Jackson, "Date Treatment of Broad Headings in 
Thirty Major Libraries: a Report with Comments," Journal of Cata - 

loging and Classification , IX (March, 1953), 21-24. 
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On the other hand it can be argued that subject headings can be 
too special, scattering the collection so much that the organizing 
potential of subject indexing is unrealized. In the extreme and 
limiting case where each different book is assigned a different 
heading, the bibliographic situation is as disorderly as if no 
heading were assigned at all. The limits of specificity have been 
remarked by Pettee: 

The present tendency of the Library of Congress 
is to use very specific terms* As the choice be - 
tween the most specific term (which can be used as 
a heading) and a more inclusive one d appends en - 
tirely upon the number of items which will be likely 
to collect under the mote inclusive term* a safe 
rule would be to prefer the mote inclusive for 
less than a dozen titles which would be likely to 
collect under it. Collecting half a dozen books 
on trees of various kinds under the term Trees, 
serves the clientele of the very small library 
better than scattering them under Elm trees. Firs, 

Hard woods, etc. The larger collection will need 
the specific terms. The choice demands judgment on 
the part of the cataloger based upon actual knowl- 
edge of his particular readers* reactions to the 
catalog,^ (my underlining) 

Similar reservations about too much specificity have been made by 
2 3 

Henry B, Van Hoesen and by Haykin, the point being there is a 
limit beyond which the number of books retrieved by any one head- 
ing cannot be sacrificed. It seems clear that an important function 
of the specific entry principle is to control what could be called 



^Pettee, Subject Headings; The History and Theory of the Al - 
phabetical Subject Approach to Books , p, 81, 

2 

Henry B, Van Hoesen, "Twelve Rules for Economy in Subject 
Headings,** an Appendix in Pettee, Subject Headings; The History and 
Theory of the Alphabetical Subject Approach to Books , p, 163, 

3 

Haykin, Subject Headings; A Practical Guide * p, 10, 
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the ’’operational breadth” of subject headings. Vigilance is nec- 
essary in keeping down (or up) the number of catalog cards piling 
up behind one heading. 

The Definition of Specificity 

The last section of this Chapter has discussed the function of 
the specific entry principle and attempted to single out the fac- 
tors which over time have proved its usefulness. The question to be 
considered in the present section is whether there is some common 
ground for understanding the concept of specificity. The answer is 
approached by elaborating seven different types of specificity, or 
specificity relations, on which one might expect to find more or 
less agreement. While this may be regarded as an exercise only in 
language analysis, it is perhaps exactly what is called for in try- 
ing to understand a concept generally agreed to be relatively vague, 
amorphous, etc. Each of the seven definitions of specificity repre- 
sents only a partial explication of what is meant by the concept, 
but each does something towards clearing part of the confusion. 

To begin with, it can be observed that it is often not the con- 
cept per se which causes trouble. The abstract concept of specifi- 
city is after all somewhat of a reification. The trouble is more 
particularized. Disagreement arises in special cases where it is 
difficult to decide whether one word is more specific than another. 
The relation of specificity is more specific than b) may or may 
not hold between two words (strictly; between two senses of meanings). 
For instance, dog is more specific than domestic animal , while dog 
and icebox , as far as any specificity relation goes, might well be 
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considered to be incomparable. It happens, but it is not so com- 
mon, that the specificity relation holds between two different senses 
of the same word. Algebra , denoting a special kind of mathematical 
system (*'a ring whose elements can be multiplied by elements of a 
specified field in accordance with special rules**) is more specific 
than algebra , meaning the modern algebra which treats of the deduc- 
tive properties of all such mathematical systems. The problem of 
specificity can be perceived then in terms of specificity relations. 
Given two words, a and b, it can be asked how much consistency is 
there in judgments of the type: a is more specific than b; that is, 

how unambiguously is the specificity relation defined. 

i. There is a sense in which the specificity relation can be 
defined completely and unambiguously. This is to regard it as the 
mathematical relation of inclusion. As the inclusion relation is 
normally defined, it is reflexive, antisymmetric and transitive.^ 

The relation is understood as holding between classes or sets of ele- 
ments, which makes possible another definition, this one in terms 
of set membership: one class A is included in another class B if 
all the members of A are also members of B. A point of some in- 
terest to note is that in the abstract theory or calculus of classes 
it is always possible, given a non-empty class, to find a different 
class which is included in this class. Given any non-empty class 
one may separate off those members which satisfy some condition and 



^These conditions are formulated as follows: 

ref lexivity g 

antisymmetry A & B ^ ^ 

transitivity 

See Patrick Suppes , Introduction to Logic (Princeton, N.J. 
Van Nostrand, 1957), Chap. 10. 
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form the class consisting of just those members. For instance, 
knowing that the class of mammals exists, one can assert the exist- 
ence of a class of mammals which have the further property of being 
reasonable. Thus the property “reason” enables man to be separated 
from mammals in general. The example is a little misleading since 
in a formal mathematical language, such as the theory of clanses, 
there are no "meanings" attached to the elements of the language; 
classes and their members are represented by letters, not by names 
such as " mammal " or ' *man . ” The theory of classes is an abstract 
language, yet it is one in which there can be no doubt at all as to 
what is meant by inclusion, whether one class is included in another. 
Unambiguously defined by postulates, the inclusion relation is in- 
dependent of "meanings" in the ordinary sense; it is very much above 
exception or disagreement. Specificity defined in terms of the 
mathematical relation of inclusion could be called "formal speci- 
ficity." Formal specificity is too precise to have more than a 
very limited application, but it is important because it can be seen 
to represent or model in a pure, idealized form other more interest- 
ing (and more confusing) types of specificity in ordinary language. 

ii. In meaningful or colloquial language the inclusion relation 
is often used with nearly the same precision as in an abstract math- 
ematical language; in these uses it satisfies the same properties of 
reflexivity, antisymmetry and transitivity. Presumably in all nat- 
ural languages there are some words which, more appropriately than 
others, can be characterized as class terms. These are the words 
which quite naturally lend themselves to extensional definition. The 
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extension of a word covers all the objects denoted by the wordg 
Cat , for instance, can be defined as the class consisting of all 
cats. The definition is understood by forming the class in a men- 
tal way, that is, by abstracting from all things a common charac- 
teristic, in this case ”catness." Similarly one can form the class 
of domestic animals , and in the process of abstraction it is seen 
that all those things characterized by catness are domestic animals 
as well. It would be fairly agreeable then to conclude that cat 
is more specific than domestic animal . Only "fairly*' because one 
might pause to reflect, for instance, that a panther can be called 
a cat but not a domestic animal i It is possible that the extension 
of any so-called class term can be toyed with, given sufficient 
imagination. But there are even more confusing difficulties when 
trying to make decisions about class membershipi For instance, are 
nectarines peaches? The dictionary has it so, but then goes on to 
point out that "nectarines may arise from peaches or peaches from 
nectarines." Taking these reservations as limiting but not contro- 
verting, it does seem that there ix a large domain in ordinary lan- 
guage where the use of the specificity relation follows closely the 
rules of logic. This is where words can be extensionally defined 
in a relatively clear way. 

There is trouble at once, however, when this concept of speci- 
ficity is extended beyond its legitimate domain to other parts of 
language, for instance to abstract words such as good , true , and 
beautiful . It is difficult to define these words by extension, One 
of the characteristics of words which can be extensionally defined 
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is that their meanings can be taught ostensively . ^ That is, it 

is possible to demonstrate the meaning of cat simply by pointing 

to different cats, however many cats are necessary to give someone 

the inductive ground for establishing an association between the 

word and the thing. Abstract words, on the other hand, are not so 

strongly referential. Only in a Platonic heaven can the good, the 

true and the beautiful be pointed to. It can be asked if there is 

a relation of specificity holding between these abstract words. The 

question is not silly, it seems a consequence of the demand for 

grounding abstract words, the demand for reference posts to make 

meanings more precise. It proved a source of difficulty for Plato 

who believed, and did not believe, that the idea of beauty, as well 

as the idea of truth, "participated in" (was included in) the idea 

of the Good. Common sense rebels. It seems as though "specificity" 

has got out of hand here -- testimony perhaps to the dictum that 

2 

"philosophical problems arise when language goes on holiday." The 
relation of class inclusion is a compelling one and attractive. 
Classifying is a fundamental operation of the mind, fundamental to the 
ordering of thought and experience. But there can be too much order. 
The meaning of the inclusion relation is fairly precise, fairly close 
to its mathematical meaning, when it is used between words which 
are clearly referential, but as its use is gradually extended to the 
more heady realms of language, its meaning becomes more diffuse, 

^See Williard van Quine, From a Logical Point of View 
(Cambridge, Mass.: Harvard University Press, 1953), p. 65 ff. 

^Wittgenstein, Philosophical Invest igations ^ p. 19. 
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more vague, more subject to the dispute of viewpoint. 

iii. There is, however, one extension of the inclusion rela- 
tion into the domain of nonreferent ial language which seems more 
warranted than others because of the exactness and simplicity with 
which the boundaries of this domain can be drawn. In effect, the 
ambiguity of the relation is controlled by stating a rule which 
makes it possible to see immediately when the relation holds. This 
rule can be seen as an extension of the operation mentioned earlier 
(p. 25) which allows one to form a subclass of a class by intro- 
ducing a distinguishing characteristic -- mammals plus aason, for 
instance, gives reasonable mammals or men« This separating operation, 
used in the abstract uninterpreted language of classes, may be 
taken as a syntactic analogue of one of the most basic semantic 
operations employed in ordinary language, namely the operation of 
modifying. The analogy perhaps is what reinforces the impetus to 
extend class inclusion beyond words which are clearly referential 
to words of a more abstract nature: from, say, reasonable mammals 

to atom: c physics . It is fairly easy to "picture" the subclass of 
mammals that are men; it is not so easy to grasp in a conceptual 
way the elements which form the subclass of physics that is called 
atomic physics. On the other hand, no one would dispute that atomic 
physics is a special kind of physics, and thus is more specific 
than physics. For the most part modification leads to specification. 
"For the most part" because unfortunately the operation is not al- 
ways well-behaved. Anomolies can arise. For instance, it is diffi- 
cult to regard metaphysics, in the sense of "supra-physical" as 
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being specific to physics. The trouble is that not all modifiers 
pare down the meaning of the words they modify. A particularly 
troublesome case -- and again one of Plato’s worries -- is that of 
negative modifiers (eg., not ) which exploit the words they modify 
in such a curious way# These modifiers are perhaps not numerous j 
and at first sight it would seem that they can be rather easily 
identified. One might thus relegate them to a stop list, so that, 
with clear exceptions, the rule would still hold that a modified 
word is more specific than the same word unmodified. This rule 
deals with pairs of words (strictly: word phrases) between which 

the relation of semantic specificity holds. It singles out and 
isolates a special kind of hierarchy. This has been called ’’phrase- 
length hierarchy” by Swanson and has been studied earlier on the 
GLS Indexing Project.^ One can without misunderstanding also call 
it ’’phrase-length specificity.” The great advantage of phrase- 
length specificity is that the specificity relation is defined in 
a thoroughly clear-cut way, independently of intuition and — assum- 
ing a stop list of troublesome modifiers -- in no way counter to it. 
The price paid for the disambiguation is that the domain in which 
the relation holds seems too partial, at least as the specificity 
relation is usually understood. For instance, it does not hold be- 
tween the words man and mammal # It is difficult to get some idea 
of the extent of this partialness. In the context of the retrieval 
of technical information, it is perhaps not very usual to find mod- 
ified noun phrases called by other names, for instance the way 

^Rayward and Svenoni ’Consistency, Consensus Sets and Random 
Deletion,” p# 4# 
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rational mammal is dubbed man * In any case there is a significant 
amount of straight modification'; and thus there is a significantly 
large and well-bounded area of language where **specificity*' can be 
said to be viewpoint free. 

iv. In the language of mathematics and in that of natural 
language where meanings are clearly referential or are related by 
easy modification, the specificity relation is neither relative, 
nor vague, nor ambiguous. The precision in each case, however, is 
gained by limiting the domain in which the relation of specificity 
can legitimately hold. But how useful are these definitions when 
it comes to making decisions in the structuring of library classi- 
fication schemes intended to map extensive areas of language, per- 
haps language as a whole? ^ Semantic tree structures, such as li- 
brary classification schemes, subject heading lists and thesauri 
represent partial orderings of words; the relation which accomplishes 

the ordering is implicitly, even explicitly sometimes, modeled on 

2 

the ideal logical relation of class inclusion. Quite often the 
structure of the subdivision of a library classification does cor- 
rectly express class inclusion — but very often it does not. 
Fairthorne gives an *'excrutiating” example where the condition of 
transitivity is violated: 

^The Dewey and L.C. classifications might be understood as 
mapping all of language, since they are intended to handle books on 
every subject. 

2 

Thus in his An Introduction to Librao; Classification , Sayers 
writes : 
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here is an excrutiating example from the U.D.C. 

681.1, Apparatus with Wheel Mechanisms, includes 
681.14, Calculating and Adding Apparatus, which 
includes 681.143.2, Slide Rules. Slide Rules 
have taken weird shapes in their time, but they 
are not apparatus with wheel mechanisms. Some- 
body forgot that not all calculating apparatus 
has wheels or, more likely, did not bother to 
look at the main heading.^ 

It seems unlikely that someone "forgot” or "did not bother to look 
at the main heading" since it is not difficult to find other simi- 
lar "mistakes" in classification schedules -- mainly because dif- 
ferent principles of division are used to form subclasses down the 
line, partly because headings are just placed somewhere because 

there is nowhere else to put them. Subject heading lists are also 

2 

to some degree structured on the relation of inclusion. It is in 



It is clear that in a classification difference is 
the determining factor of division itself. Likeness 
makes the genus, i.e., draws all the species or all 
the things having it into one family; difference is 
the something added to these things which abstracts 
the species or members of the family. In outline, our 
classification of man shows that 



Genus 



Difference 



Species 



zoology 

animal 

vertibrate 

mammal 



sensibility 

backbone 

viviparous quality 
reason 



animal 

vertebrate animal 

mammal 

man 



Every classification must prodeed on some such lines as 
these. 

From W.C. Berwick Sayers, An Introduction to Library Classification 
(8th ed.; London: Grafton, 1950), p. 27. 

^R.A. Fairthorne, "The Mathematics of Classification," Towards 
Information Retrieval (London: Butterworths, 1961), p. 6. 

2 

Pettee writes: "Under the particular topical heading all 

aspects of the topic may be collected. These particular topics 
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the "see also" structure of subject heading lists that the inclusion 

relation is reflected.^ And again it is used imperfectly, though 

in this case at least, the deviation from strict inclusion is both 

conscious and intentional. It is also a matter over which there 

2 

has been considerable disagreement. The problem is whether the 
directioning of "see also" references should be reserved for going 
from general to specific, or whether they should be used also to 
indicate coordinate relationships. It is a problem of deciding 
between a rigorous inclusion (specificity) relation or a more flex- 
ible, but vaguer, relation of association. The problem is somewhat 
academic, since in practice the decision is already made -- in favor 

3 

of the liberal alternative. In any case it is clear that there 



dispersed through the alphabet must still be considered parts of 
a larger whole with interrelationships to many allied topics. An 
alphabetical subject catalog as well as a classed catalog must take 
into account these interrelationships, and to determine them a lo- 
gical analysis of the topical groups is necessary." 

Pette, Subject Headings: The History and Theory of the Alphabetical 

Subject Approach to Books , p. 59. 

^In the introduction to Sears List of Subject Headings one 
reads; "In general, *see also' references are made from the gen- 
eral subject to more specific parts of it, and not ordinarily from 
the specific to the general." 

Sears List of Subject Headings ,, ed. Bertha M. Frick (8th ed.; New 
York: H.W. Wilson, 1959), p. 25. 

"Not ordinarily," but sometimes, and the reference in both directions 
violates another of the conditions used to define the relation of 
inclusion, viz . , that of antisymmetry. In fact the very first entry 
in Sears List that uses a "see also" shows a downward reference 
from abbeys to convents and nunneries--monasteries . as well as a 
reference from convents and nunneries--monasteries to abbeys . 

^See Frarey, Subject Headings , p. 42 ff., p. 6^^ ff. 

3 

In rationalizing the decision, Pettee on the structure of sub- 
ject catalogs writes: "The logic transcends the limits of a classi- 
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are more relations in both a library classification and a subject 
heading structure than can be expressed by any strict theory of 
inclusion. One might say that the specificity relation when it 
is used irregularly or imprecisely -- as in the slide rule example -- 
becomes unmoored from its logical foundation, and is, thus, cor- 
rupted. Though it is perhaps truer to say that a mathematically 
precise concept of specificity is not so much corrupted as simply 
found wanting, when it comes to mapping a larger area of language. 

But there is an alternative precision to the mathematical pre- 
cision which is achieved by the setting up of postulates or condi- 
tions. A relation, such as the specificity relation, can be defined 
exactly by an explicit enumeration of all the pairs of objects (words) 
for which the relation holds. In other words, a consensus of judg- 
ment about specificity (slide rules are more specific than apparatus- 
wheel mechanisms) need not derive from reason or rules of logic. A 
consensus can be obtained by force, and this seems to be what happens 
in practice. The structure of a subject heading list such as Sears 
must be taken as given; hierarchy is what its authors choose to 
call hierarchy, the relation of specificity is defined by enumeration. 



fication scheme, for the interrelationships of the special topics 
reach out into the whole field of knowledge • • • The interrela- 
tionships brought together under names in the dictionary catalog - 
are impossible in the logical sequence of strict classification 
lines." (Pettee, Subject Headings; The History and Theory of the 
Alphabetical Subject Approach to Books , p. 59.) 

As was indicated earlier, classification schemes are not particular- 
ly limited by considerations of logic. But this is a matter of 
degree, and Pettee* s observation is probably good as a black-white 
picture of the emphatic difference between subject heading lists 
and classification schemes. 

^See Elaine Svenonius, "Library Classification Lattices," 
Master’s Thesis (Chicago: University of Chicago, Graduate Library 

School, 1965) . 



40 



34 



There can be no disagreement about the relative specificity of two 
subject headings, since one can see whether a specificity relation 
holds in one direction, in both, or not at all, simply by consulting 
the List to see where in the established hierarchy the headings are 
located# The List is very definitely an authority list, whose pur- 
pose is to impose standardization, to curtail the cataloger’s rights 
of imagination for the sake of uniformity. Catalogers are cautioned 
about departing from the List , even in the eventuality, which seems 
probable enough, that a **new" subject emerges.^ 

This juggernaut quality of the List has quite naturally been 

cause for complaint, though usually authority lists do leave room 

2 

for some flexibility in the use of the terms they contain# The 
permissiveness allowed, however, is not so much in judging what is 
specific to what, but rather in choosing the degree of specificity 
at which a particular library wishes to operate. (As will be sug- 
gested later, as concerns libraries this is perhaps the proper con- 



"Tentative headings can be assigned, perhaps written in pen- 
cil on the cards to be used until the terminology becomes stand- 
ardized# A list of these tentative headings should be kept (it 
will never be long) so that they can be reconsidered later and 
either adopted permanently or changed, as the case may be, and 
added to the list." ( Sears List of Subject Headings ^ 8th ed., 
p. 27.) 

2 

In the Preface to the ninth edition of Sears , Barbara Westby 
writes: "A common criticism of any list concerns the degree of 

specificity in its headings. Specificity is relative and depends 
on the size of a library, its function, and its patrons. In a 
small collection, the use of too specific headings can result in 
scattering like materials. Practicality rather than theory should 
determine the degree of specificity. Sears, by example or instruc- 
tion, suggests 183 classes of headings that may be added by the 
cataloger. The provision of 126 subdivisions further increases the 
degree of specific entry a library may use." (Sears List of Sub- 
ject Headings (9th ed#: New York: H.W# Wilson, 1965), p. 6.) 
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text in which to discuss the relativity of specificity, one where 
the problem is real.) Not only is there some permissiveness in an 
authority list, but the government it imposes is not entirely ar- 
bitrary. Terms used for subject headings must conform to current 
American usage: 

As a general rule, however, when a term sanctioned 
by American usage is available for an object (or 
group of objects), a concept, or a relationship, 
it may be used as a subject heading. Under this 
principle there is justification for such specific 
headings as Autumn > Quatrain . Ultrafax . Waltz . 

In fine, a subject heading used in the modern 
dictionary catalog or alphabetical subject catalog 
represents a choice of that term to designate the 
subject which is to be used consistently regard- 
less of the language of the author. This term 
must, if possible be sanctioned by current American 
usage. ^ 

The making of an authority list, like the making of a dictionary, 
is not a willful or whimsical undertaking. It happens sometimes 
that a dictionary definition does not do justice to the contextual 
subtleties of usage of particular words, but, whether one accepts a 
definition or not, one still uses a dictionary. If a library user 
does not consult a subject-heading list, then possibly it is be- 
cause he already has enough familiarity with the bibliographic 
mapping of the collection or he has found that that mapping is 
close enough to a use of language with which he is familiar. Any 
difficulties that arise because of a divergence between his own 
language and that of a library can in part be resolved by a simple 
consultation with a subject headings list. The most compelling 
reason for coercive specificity is that the consistency it creates 



^Haykin, Subject Headings: A Practical Guide , p. 11. 
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furthers the chances for a dialogue between the catalog and a 
library user. 

V. Uriel Weinreich writes that one of the major motivations 

of semantic research is **a desire to analyze global meaning into 

1 2 
components.” It has been suggested by Thyllis Williams that a 

quantitative measure of semantic specificity be based upon the re- 
sults of this kind of componential analysis. In developing the 
idea, Miss Williams focuses her attention on a special class of 
word meanings, viz , , meanings which are ^commonly accepted” in the 
sense that they can be found in a standard pocket dictionary. A 
thesaurus constructed of hierarchies of these word meanings can be 
obtained by turning a dictionary ”inside out,” that is, by subordinat- 
ing every defined word (strictly: word sense or meaning) to each 

of its defining words. Roughly, the specificity of a word is pro- 
portional to the complexity of its definition, Man is more specific 
than mammal because to the definition of mammal one must add another 
component, viz . , the distinguishing characteristic "rational.” In- 
tuitively it would seem that there must be some correlation between 
the specificity of a word and the number of components used in its 
dictionary definition. It is apparent at once, however, that cer- 
tain qualifications must be made. First, not all the components in 

^Uriel Weinreich, "Explorations in Semantic Theory,” Univer - 
sals of Language , report of a conference held at Dobbs Ferry, N.Y., 
April 13-15, 1961, ed. Joseph H. Greenberg (Cambridge, Mass: 

M.I.T. Press, 1963), p. 405. 

2 

Thyllis M. Williams, "Standardized Abstracts of Dictionary 
Definitions,” Studies in Indexing Depth and Retrieval Effectiveness 
(NSF GN 380/654) (Chicago: University of Chicago, Graduate Library 

School, 1968). 
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a dictionary definition are equally significant. For instance 
some prepositions and other common words are not particularly de- 
scriptive and they therefore contribute little to the meaning of the 
word being defined. The problem becomes more complex when one 
realizes that there is no sharp boundary between woros which are 
descriptive and those which are not; words are only more or less 
descriptive. Secondly the syntax of the definition must be consid- 
ered* Words which are used disjunctively in a definition (eg., 

"new" as defined as “recently discovered, recognized or learned 
about") can be regarded as contributing essentially less to the 
definition than words not so used ("inquiry" defined as "a request 
for information"). More examples could be given and of a more com- 
plicated nature, the point being that the analysis of dictionary 
definitions requires thoughtfulness and care. Consistency requires 
that such an analysis follow a well-defined set of rules. The re- 
sult of applying these rules to a dictionary definition is called 
a "definition abstract." The definition abstract consists of the 
components, of the definition selected as significant, and a 
weighting of these components according to their significance. 

Having thus obtained definition abstracts of words, it would 
seem that the specificity of the words could then be measured by 
counting significant components. But even this is too simple be- 
cause the complexity of the components themselves must be taken 
into account. There is reason to hope, however, that by carrying 
the process of abstraction through level after level of definition 
(i.e., abstracting definitions of definitions) one will eventually 
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approximate a set of atomic compcaents, what might be called the 
"basic English of definition." It is at this point then that the 
specificity of a word could be measured by counting the number of 
at:>mic components needed for its definition. This work on compo- 
nential specificity is not finished yet, but continues to be carried 
out in increasing detail and sophistication by Miss Williams. 

There are two advantages to the concept of componential spe- 
cificity. The first is that by going beyond taxonomy it can be used 
to provide a measure according to which all words are comparable 
with respect to specificity. For instance it makes the specificity 
of slide rule less than, equal to, or greater than that of eternity . 
This may be somebwat unintuitive, but it is at least as acceptable 
as the "atomization of meaning" and the semantic tradition to which 
it belongs. The second advantage is that, when one approaches spe- 
cificity in this way, use is made of a great body of scholarly re- 
search on meaning, viz . , the dictionary. Such an approach is firm- 
ly objective about meaning and meaning relationships in natural 
language, more objective, say, than that seen in the quick, ex- 
pedient and often ad hoc construction of special purpose thesauri. 

vi. So far it has been seen that the ambiguity of specificity 
is diminished when it is understood in terms of a strict inclusion 
relation, "strict" in the sense of the relation having a limited 
and well-defined domain, either in mathematical or ordinary lan- 
guage, or when each of the special cases of the specificity rela- 
tion is enumerated in the form of an authority list or a thesaurus 
which is more or less sanctioned by ordinary language. Another 
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approach to finding some common denominator in specificity-view- 
points is to study directly the common usage of language, in partic- 
ular what might be called the "sociology of classification," Ap- 
propriate to such a study are socio-linguistic experimental methods, 
especially those making use of questionnaires. The objective would 
be to discover and measure the amount of consistency to be found 
in different people's opinions about specificity. An experiment 
might be designed, for instance, wherein people are asked to order 
words on the basis of their intuition or immediate perception of 
specificity. The results could be expected to indicate how regular 
the specificity relation is from the point of view of shared lan- 
gua^:e behavior, A possible difficulty with such an experiment is 
that there are many instances where common sense would balk at 
having to make a decision about a relationship; for instance, shoe 
and eternity night naturally be regarded as being outside the proper 
domain of a specificity relation, i,e,, as being incomparable. To 
force an ordering where none seems immediately apparent could re- 
sult in some fairly arbitrary classification patterns, with little 
chance of overlap. On the other hand an experiment need not be 
so blunt, and more subtle studies of language behavior using ques- 
tionnaires would be useful in bounding the domain where there is 
consensus as to the "correct" use of the specificity relation, 

vii, A seventh type of specificity is one which seems partic- 
ularly applicable to language as it is used in indexing the col- 
lection of books or documents in a library. This specificity has 
been referred to on the GLS Indexing Project as "operational spec- 
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ificity or simply, "breadth, ” The term "operational" is used to 
distinguish this type of specificity from any of the preceding 
types, which may be loosely referred to as "semantic" specificity. 
The operational specificity or breadth of an index term or sub- 
ject heading is defined as the number of items in the collection 
indexed by the term, that is, the number of postings made to the 
term. The average breadth of indexing for a given collection is 
the number of documents indexed by an "average" term, or the total 
number of term postings divided by the number of terms in the vo- 
cabulary used to do the indexing. In other words, the breadth of 
a term is its frequency of occurrence. Average breadth is average 
frequency of occurrence. 

There are several points to be noted about this definition of 
breadth. The first is that the definition is an extensional one, 
the extension in question being an extension of a particular and 
very definite kind. As was indicated earlier the extension of a 
word in ordinary language is its denotation, or the range of ob- 
jects to which the word may be applied. Analogously the extension 
of an index term may be considered to be the books in a collection 
to which the term is assigned. There is some precedence in the lit 
erature of indexing for regarding an index term as denoting a class 
of documents : 

The index term 'Newfoundland* refers , therefore, to 
the class of documents in which the individual 
entity 'Newfoundland* is discussed. That such a 
class (or group) of documents exists is indis- 
putable, That the further use of the index term 
'Fauna* would serve to describe another class of 
documents is equally true, and the conjoint of the 
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two would designate a possible -class of documents 
in which the fauna of Newfoundland were discussed, 
even though no example of the class were immediately 
available in the collection.^ 

A class of documents or books is the referent of an index term, 
it is what is ’’indicated” by the indicating function of the term, 
and it is what is retrieved by the term in an information retriev- 
al operation.. . . in short, it is an extensional or operational 
meaning of the term. Within a given collection the number of 
items in such a class can be counted, and thus one is provided 
with methods for obtaining the quantification necessary for a 

theoretical or empirical study of the effect of breadth (specifi- 

2 

city) of term on retrieval performance. 

The definition of operational breadth makes use of statis- 
tics — the statistics of an indexed collection of documents -- 
rather than semantics. As was just seen, however, there is an 
analogy between at least one kind of semantic meaning, viz , , de- 



Center for Documentation and Communication Research, School 
of Library Science, Western Reserve University, ’’Comments on ’A 
Logician’s Reactions’,” American Documentation , VIII (April 1957), 

pp, 120-121. 

2 

Breadth or operational specificity as it has been defined 
is understood to be a property of an index term, viz . , the number 
of documents to which the term is assigned. One can also speak of 
a relation of operational specificity. A strict interpretation of 
the relation would be to regard it simply as the inclusion rela- 
tion holding between classes whose members are books. Thus if all 
books indexed by fluid mechanics are also indexed by mechanics , 
then fluid mechanics is in specificity less than or equal to mechan- 
ics. The inclusion relation is the strict one discussed earlier, 
having the properties of transitivity, reflexivity and antisym- 
metry. A weaker interpretation of the relation of operational spe- 
cificity can be obtained by relaxing the condition of antisymmetry 
which makes it possible then to say that one term is less specific 
than another if it indexes fewer books. The weaker interpretation 
has the advantage that all terms become comparable with respect to 
specificity. 





42 



notative or extensional meaning, and the meaning of an index term 
when it is regarded as referring to a class of books. But it is 
an analogy only, and must be regarded as such. The difference is 
that in one case the objects denoted by words are things in the 
real world, in the other, books in a library collection. It is 
perhaps interesting to ask how far the analogy can be pushed, how 
much correlation there is between operational and semantic (ex- 
tensional) breadth. One of the objections that can be advanced 
against the definition of operational specificity is that it is 
counterintuitive in the sense that if in one library there are 10 
books indexed by Dog and only five by Mammal , then mammal is more 
specific than dog. The specific entry principle precludes the use 
of a suitable broad term when another suitable but more specific 
term might better describe a book. This has the effect -- desired 
-- of facilitating searching and retrieval by preventing too many 
entries from accumulating under any one subject heading. And it 
can quite easily happen that there are fewer entries under Mammal 
than under Dog , This example would suggest that there is, in 
fact, no correlation at all between semantic and operational 
breadth. It can be argued, however, that a correlation can be seen 
if one takes into account implicit entries in the catalog. That 
is, if all books on dogs were counted as though they were indexed 
under Mammals > then there would be ten books on dogs and fifteen 
on mammals — and the semantics would be straight. The semantic- 
operational breadth correlation can be superimposed in a more for- 
mal way by introducing a thesaurus together with a rule which says 
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that if a book is indexed by a semantically specific term it is 
automatically regarded as being indexed by any more inclusive 
term. According to this rule the class of books on dogs would be 
regarded as being included in the class of books on mammals. A 
retrieval search is broadened by replacing the search term Dog 
by the term Mammal . The semantic breadth of the search term is 
increased here; within any given collection, there must be, quite 
automatically, a corresponding increase in operational breadth, 
so long as the collection contains some books on mammals that are 
not also dogs. Artfully then, a correlation can be established be- 
tween semantic and operational breadth. 

Though it is true there is some correlation between semantic 
and operational breadth, it is important to be aware of the dif- 
ferences, as these are important in understanding the operation of 
an indexing system or a library classification. The question of 
correlation is a particular formulation of a wider question about 
the relationship of an indexing language to language in general. 

An earlier approach to this question was to distinguish between 
natural and artificial classification,^ and then to ask: does a 

library classification reflect the structure of knowledge (infor- 
mation) -- and perhaps in so doing describe a natural order of 
things; or is the ordering simply a convenient and somewhat arbi- 
trary ordering of books? The distinction here is between biblio- 
thecal and scientific classification, between words being used to 

^See W.C. Berwick Sayers, A Manual of Classification for 
Librarians and Bibliographers , 2nd ed. (London: Grafton, 1944) , 

esp. Chaps. II and IX. 
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indicate classes of entities, and words used to indicate classes 
of books. Superficially the language thct orders books looks to 
be the same as the language that orders the world. But it is im- 
possible to imagine a complete isomorphism between books and things. 
The same language used for two different purposes is not exactly 
the same. A word used as an index term or a subject heading is 
not the same as itself when it is used to describe something in 
the world. In Wittgenstein's terminology, the one word belongs to 
two different "language games," and between these two games there 
are similarities as well as differences. And between semantic 
and operational breadth there is and there is not a correlation. 

A seeming disadvantage of the definition of operational spe- 
cificity is that it makes the breadth of any term relative to a 
particular collection of documents. It can happen, for instance, 
that a given term is broad in one collection and quite narrow or 
specific in another, depending upon the nature of the collection 
and the public for whom it is indexed. But it can be argued 
that a definition that allows breadth to be relative is in fact 
quite accommodating in that it reflects a fairly accurate picture 
of just what is the case. Different libraries find it expedient 

to operate at different levels of specificity. It is generally 
2 

assumed that the larger the library the more specifi^ must be 

^See the reference from Sears , cited earlier, p. 34. And 
Lilley writes: ". . . if the same book is described satisfactor- 

ily at different levels of specificity depending on the library 
collection to which it belongs, it can be said that 'specificity 
is in part a function of a particular library'," (Lilley, "How 
Specific Is'Specific' ? '* p. 5). 

2 

See for instance Pettee, Subject Headings: The Hi story 

and Theory of the Alphabetical Subject Approach to Books . 
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the subject headings It uses# One library, for instance, may 
have 1000 books in its collection appropriately headed Printing , 
while another may have only 10 books on this subject. For ease 
of reference the first library may choose to subdivide its books 
on printing, for instance into geographic categories, Printing»U,S, , 
Pr inting*France , etc# Ease of reference would not require this 
in the second library. The relativity of specificity perhaps loses 
some of its disturbing quality if the question "How specific is 
specific?" reads "How precise is precise?" For one rather expects 
precision to be relative# At least the rejoinder "precise enough 
for what?" seems acceptable# In the context of assigning headings 
to books in a collection, the question "specific (precise) enough 
for what?" can be answered, "specific enough for easy reference#" 

It seems important to distinguish between questions about the 
meaning of "specificity" (how is the specificity relation under- 
stood) and the question: how specific (precise) must a description 

be to be useful for some purpose, for instance for indexing a book 
in a particular library^ It is the latter question which most 
properly engenders comments about the relativity of specificity, 
since there is no single description which from a global point of 
view is correctly specific, there are rather many different de- 
scriptions made to suit different purposes or requirements# It is 
in a context such as this that a definition of operational specifi- 
city is useful# Operational specificity is decidedly relative, but 
it is clearly and unambiguously so# Its relativity reflects the 
very legitimate variabilit}^ not of "specific," but of "appropriately 

/ 
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specific, specific enough for some purpose.** 

Finally the definition of operational specificity makes ex- 
plicit a concept of specificity used in natural language, in par- 
ticular **specif icity** as it is used in the application of the spe- 
cific entry principle. As was seen^ an important function of the 
specific entry principle is to insure that not too many entries 
accumulate under any one heading, a situation that would make 
searching tedious, that would detract from **easy reference.** With 
observation that it is also possible to err in the other direction 
and to have too much specificity with too few entries under one 
heading, the intent of the specific entry principle is clarified 
in a sign* 'icant way. It can be argued that it was never intended 
to mean **be as specific as possible,** the intent was rather to be 
relatively specific, to **be only as specific as necessary.** 
Pettee*s remark might be repeated here: 

As the choice between the most specific term 
(which can be used as a heading) and a more 
inclusive one depends entirely upon the number 
of items which will be likely to collect under 
the more inclusive terms, a safe rule would be 
to prefer the^more inclusive for less than a 
dozen titles. 

In other words, the operative factor in determining the specif- 
icity necessary is the number of items posted to a given beading, 
viz , > the operational breadth of the heading. 

^See pp. 20,’ 21 and 22, 

^See p, 22, 
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Summary 



Wittgenstein writes that ‘^philosophy is a battle against the 
bewitchment of our intelligence by means of language.**^ As a 
method of approach, the philosophical one seems appropriate in 
dealing with the confusion which has centered around the concept 
of specificity, a concept of special importance in the theory of 
indexing. The method demands the making of distinctions. The 
question which begins this paper is “How specific is specific?" — 
a somewhat vague question, and largely rhetorical, intended only 
to underscore the fact that "specificity" is viewpoint dependent. 
Though the concept of specificity may have vague boundaries, there 
is a large area in the center which is not vague -- that is, there 
are many particular cases where there can be no viewpoint-gen- 
erated misunderstanding at all as to its meaning. A simple illus- 
tration of this is to observe that the following classification 
can be funny: 

In this text, Borges quotes 'a certain Chinese 
encyclopedia where it is written that "Animals 
are divided into a) belong to the Emperor, b) 
embalmed, c) tamed, d) suckling pigs, e) sirens, 
f) fabulous, g) dogs at liberty, h) included in 
the present classification, i) which act like 
madmen, j) innumerable, k) drawn with a very 
fine camels hair brush, 1) et cetera, m) which 




The main thrust of the present chapter has been to coioider 
what common ground there is for understanding the concept of spec- 

^Wittgenstein, Philosophical Investigations ^ p, 47. 

^Quoted by Richard Poirer, "The Politics of Self Parody," 
Partisan Review , XXXV (Summer 1968), 352. 

O 
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ificity. Seven different types of specificity or specificity 
relations on which one might expect to find some agreement were 
discussed. They are, in summary: 

1) Formal Specificity ; Specificity can be defined in terms 
of the logical relation of class inclusion. One class A 
is more sp^^fic than another class B if and only if A is 
properly included in B. Formal specificity is an abstract 
concept and is divorced from the **meanings** of ordinary 
colloquial language. Much of the apparent relativity of 
semantic specificity can be understood as resulting when 
formal specificity is extended in imperfect and irregular 
form beyond its legitimate domain to the whole of lan- 
guage. 

2) Extensional Specificity ; In ordinary language the spec- 
ificity relation (regarded as inclusion) is used with 
logical precision when it holds between classes that can 
be clearly defined in extensional terms. There specif i<^* 

V 

ity is not relative; no one would disagree that cat is 
more specific than mammal , but there would be little 
agreement that beautiful is more specific than good . 

3) Phrase Length Specificity ; One extension of the inclu- 
sion relation into the domain of nonreferential language 
seeMS fairly unambiguous. This is when specification is 
regarded as modifying. There are exceptions, but gen- 
erally it holds that a word modified is more specific 
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than a word unmodified. Red house is more specific than 
house . 

4) Coercive Specificicy ; The specificity relation can be de- 
fined more or less well by enumerating all the pairs of 
objects (words) between which the relation is supposed to 
hold. In effect this is what is accomplished by the con- 
struction of thesauri, classification schedules and author- 
ity lists. There can be no disagreement in practice about 
the relative specificity of two subject headings; it is 
simply a matter of consulting an authority list to see 
whether a specificity relation holds between them, in one 
direction, in both or not at all. 

5) Componential Specificity : A quantitative measure of spec- 

ificity has been developed by Thyllis Uilliams. Roughly 
the specificity of a word is proportional to the complex- 
ity of its dictionary definition, where definition com- 
plexity is understood in terms of both the descriptive 
components and the syntax of the definition. 

6) Consensus Specificity : Presumably there exists some par- 

tial consistency in different people *s opinions about 
specificity, a consensus whose bounds are unknown but 
might be measurable using socio-linguistic experimental 
methods . 

7) Operational Specificity : Operational specificity is de- 
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fined in the context of indexing, or assigning subject 
headings to, books in a library. The operational spec- 
ificity of an index term is the number of books in the 
collection indexed by the term. The definition is an ex- 
tensional one, in that the "meaning** of an index term is 
regarded as its extension, viz . , the class of books to 
which the term is assigned. Operational specificity is 
decidedly relative, but it is so in a clear, mathemati- 
cally measurable way. Its reluctivity reflects the very 
legitimate variability not of '"specific,** but of "specific 
(precise) enough for some purpose.** Further, the defini- 
tion of operational specificity goes some way to make ex- 
plicit the concept of specificity as it is understood in 
the application of the specific entry principle. It does 
this insofar as the function of the principle is to regu- 
late the number of entries that accumulate under any one 
heading. In the discussion of the uses of the specific 
entry principle it was shown that if this is not the main 
function of the principle, it is at least an important one. 
Since the operational specificity of an indexing can be 
measured and since’, as will be seen (p. 53), there is a 
method for systematically varying the operational speci- 
ficity of indexing, this definition is the one chosen for 
use in the present experimental study of the effect of 
specificity on indexing effectiveness. 
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EXPERIMENTAL DESIGN 
Introduction 

The question, ’’What is good indexing?" is vague and unclear. 

It may bo that in the long run a question like this cannot be an- 
swered in any simple way, yet it is one which has led to much spec- 
ulation. Perhaps rather necessary speculation, since the belief 
that some index terms are better chosen than others is the raison 
d * etre of indexing practice so far as it is not wantonly hap- 
hazard buc based on indexing principles and authority lists. It 
has not been until recently that speculation has given way to 
more objective criteria for evaluating different methods of in- 
dexing. Notable in the early attempts at such evaluation is the 
first Cranfield project (1960)^ which, under the supervision of 
Cyril Cleverdon, carried out experiments to test the comparative 
performance of four indexing systems. Systems performance was 
measured in terms of the amount of relevant and irrelevant mate- 
rial retrieved in response to search requests addressed to an 
indexed collection of documents. The importance of the experi- 

^Cyril W. Cleverdon, Report on the Testing and Analysis of 
an Investigation into the Comparative Efficiency of Indexing Sys- 
tems (Cranfield, England: College of Aeronautics, ASLIB Cran- 

field Research Project, October 1962). 
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TOent derives from its being the first large scale effort to use 
objective criteria for determining the relative **goodness*' of 
different indexing methods, suggesting thus that there may in 
fact be a rational way to choose from among different systems and 
their conflicting claims. 

Since 1960 further experiments, rather more scientific in 
nature, have been carried out, and other more subtle measures for 
evaluation have been proposed. At the same time, perhaps as a 
lesson learned by hindsight from mistakes made in the early 
Cranfield experiment, there has been a tendency away from the 
wholesale comparison of different indexing systems towards the 
singling out of certain individual factors involved in the index- 
ing process and seeing how variation in these affects retrieval 
performance. Here again the work of Mr. Cleverdon is notable, 
this time in the Cranfield II project, an experimental study of 
factors determining the performance of indexing systems.^ In- 
dexing depth, the number of index terms assigned to a document, 
is one of the factors affecting retrieval performance. This fac- 
tor is studied from a theoretical point of view by Swanson in 



^The report of the Cranfield II project is contained in the 
following publications: 

Cyril W. Cleverdon, J. Mills and M. Keen, Factors Determining the 
Performance of Indexing Systems . Vol. 1 — Design; Part 1 — 
Text, Part 2 -- Appendices (Cranfield , England : College of Aero- 

nautics, ASLIB Cranfield Research Project, 1966). 

Cyril W. Cleverdon and M. Keen, Factors Determining the Perfor - 
mance of Indexing Systems , Vol. 2 — Test Results (Cranfield, 
England: College of Aeronautics, ASLIB Cranfield Research Pro- 
ject, 1966) . 
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"On Indexing Depth and Retrieval Effectiveness*"^ A model of an 
indexing system is developed which makes it possible to predict 
the effects on recall and precision when indexing depth is varied 
by randomly deleting index terms from documents. 

Specificity of indexing can also be regarded as a factor that 
affects retrieval performance# The definition of operational spec- 
ificity developed in the last chapter makes experimental testing 
possible, and, as was suggested, this definition at least approx- 
imates the meaning of "specificity" as it is implied in the usage 
of the specific entry principle. And indeed the main objective 
of the present study is to test the effect on retrieval perform- 
ance of indexing specificity so defined# 

An objection to attempting to find the goodness of an index 
term in the measure of its operational specificity of breadth, 
which is a statistical property of a particular indexed collection 
of documents, is that this takes no account of the relation an 
index term bears to what it indexes. It seems natural to regard 
a good index term as one that telescopes sharply the central con- 
cepts in the work being indexed, one that in some way is seman- 
tically appropriate# The semantic appropriateness of an index 
term is difficult enough to comprehend let alone evaluate# In- 
tuition, of course, is a method that can be used to determine 
which index terms best describe the content of the works being 

^Don R# Swanson, "On Indexing Depth and Retrieval Effec- 
tiveness , "Proc£edijig£ of the Second Congress on the Information 
Systems Sciences, Hot Springs, Virginia, November 1965, pp. 311- 
319. 
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indexed. In some indexing practices this intuition is exploited 
and terms are weighted according to their felt importance to the 
text. Another method of do limiting semantically appropriate in- 
dex terms is to assume that certain words, by virtue of their 
location in positions of importance in the work being indexed, are 
more semantically descriptive than others and are, thus, the most 
suitable candidates for index terms. As was seer in the preced- 
ing chapter, since the middle of the 19th century the title of a 
work has traditionally been pointed to as the position of prime 
importance. A secondary objective of the study is to test the 
hypothesis that the quality of index terms assigned to documents, 
rather than their specificity, is the operative factor in retrieval 
effectiveness , 



Overview of the Experiment 



The Salton-Cranfield data, a subset of the Cranfield II 
data, has been borrowed for use in the experiment to be described. 
It consists of 200 documents, mainly in the field of aerodynamics, 
and 42 search questions to which the documents in the collection 
have been judged, by subject experts, as relevant or irrelevant. 
The documents have been manually indexed by trained indexers to a 
considerable depth, on the average 34.5 terms per document. What 
recommends this data for the experiment, besides its availability. 



The report of Salton’s experiments with the subset of the 
Cranfield II data is given in: Gerard Salton, ''Computer Eval- 

uation of Indexing and Text Processing,” Journal of the Associa- 
tion for Computing Machinery, XV (January 1968), 8. 
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is that it is the only data where the relevance judgments have 
been made by subject experts and where an indexing of this depth 
has been done manually by trained personnel# 

The general testing procedure is the following. The indexing 
of the documents is systematically varied by deleting certain of 
the index terms. For instance in one operation broad terms are 
deleted from documents ; in another, narrower terms are deleted; 
index terrfts are deleted from a document if they do not also occur 
in the title of the document; or terms which are not heavily^ 
weighted, i.e., not thought especially important by the indexer, 
are deleted* At each deletion, a subset of the terms originally 
assigned to the documents in the collection is removed, the pur- 
pose being to see if this change in the indexing makes a signif- 
icant difference in retrieval performance. Thus the ^'importance” 
of different sets of terms, broad terms, non-title terms, etc., 
is judged by the effect of their absence from the indexing of the 
documents • 

After each deletion, the 200'”document collection is searched 
in order to retrieve documents responding to the search questions 
that are addressed to the collection* The searching procedure is 
of the usual type where terms used to index the question are 
"matched” with the terms used^to index the documents in the col- 
lection* The output of the searching is then evaluated according 
to different measures of retrieval effectiveness* In this exper- 
iment, the familiar precision and recall measures are used, as 
well as the Expected Search Length (esl) measure developed by 
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Cooper.^ The testing procedure then, is to delete index terms, 
to retrieve documents in response to search questions and to 
evaluate the precision-recall and esl effectiveness of the al- 
tered indexing. Each time these operations are performed, the 
question is asked whether the esl, precision and recall values 
obtained using the altered indexing differ significantly from 
those yielded by the original indexing, before any terms were de- 
leted* The statistical test used to assess the relative effect- 
iveness of the before-and-after deletion indexing is the Wil- 
coxen Test. Most of the experiment is programmed in PLl; Fortran 
is used for the Wilcoxen Test, The remainder of this chapter 
discusses in detail the variables in the experiment, the de- 
letion procedures, the measures of retrieval effectiveness, the 
Wilcoxen statistical test, the Salton-Cranfield data, and the 
progratnming strategy used in the experiment. 

The Variables 

1) Operational specificity or breadth (B) : This has al- 

ready been defined in Chapter 1. The operational breadth of a 

term is the number of documents, in a given collection, which 

2 

the term is used to index. Average breadth (B) is the number 

^William Coooer, ’'Expected Search Length: A Single Measure 

of Retrieval Effectiveness Based on the Weak Ordering Action of 
Retrieval Systems," American Documentation * XIX (January 1968), 30. 

2 

Average breadth is a property of an indexed collection of 
documents; thus, one can speak of the "average breadth of index- 
ing" or of "average indexing breadth." As it seems unlikely 
that ambiguity can result, the modifier "average" is sometimes 
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of documents indexed by an "average term," or the total number 
of term postings divided by the number of terms on the vocab- 
ulary list from which the index terms are selected. 

1 

2) Depth (D) : This variable will be defined here in the 

same sense as that used by Swanson in the paper "On Indexing 

2 

Depth and Retrieval Effectiveness." Indexing deptii is the num- 



omitted: by "indexing breadth" and "breadth of indexing" is 

meant average indexing breadth and average breadth of indexing* 
The word average is also omitted in similar expressions of in- 
dexing depth. 

^The term "exhaustivity" is common in the information re- 
trieval literature but is avoided here. Exhaustivity is a 
measure of the extent to which all the distinct "concepts" dis- 
cussed in a particular document are recognized in the indexing 
operation* It is opposed to specificity which is the generic 
level at which a concept is recognized in the indexing* Concep- 
tual analysis is philosophically a bug-bear. The trouble with 
concepts is that they are not always clear and distinct; es- 
pecially in the less hardcore sciences and humanities this is 
true. Moreover, thesauric relationships between concepts are 
often ad hoc, subject to point of view, even illogical (eg., 
Fairthorne's example from U.D.C. where slide rules are specific 
to apparatus with wheel mechanisms). Occam's razor might be 
taken to the distinction between exhaustivity of indexing and 
specificity of indexing, on the grounds that operationally the 
distinction is not real but exists by definition. Both specific- 
ity and exhaustivity can be reduced by moving to a broader 
class; in the one case by moving up in a thesaurus hierarchy, in 
the other by moving to a lower cutoff point. However, depending 
on how "index term" is defined, a move say from "finite wings" 
to "wings" could be interpreted either as amove in hierarchy 
or exhaustivity ("finite" and "wings" could be two index terms, 
or "finite wings" c:'uld be one). Indexing and searching are not 
independent operations. In the literature one sees attempts to 
separately evaluate these two operations, glossing over possible 
interdependencies. It can be argued that the difficulties arise 
owing to a terminology that is not clearly explicated* 

2 

Swanson, "On Indexing Depth and Retrieval Effectiveness," 
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^ * 

ber of index terms assi.gned to a document* Average depth (D) 
for an indexed collection of documents is the average number of 
terms assigned to a document, or the total number of term post- 
ings divided by the number of documents in the collection. 

3) Collection Size (N) : This is simply the total number 

of documents in the collection. In the present experiment N is 
regarded as a constant. 

4) Vocabulary Size (V): It is assumed that in the indexing 

of documents terms were assigned from a controlled vocabulary or 

1 

authority list of allowed possible terms. Vocabulary size is 

then defined as the number of terms on this vocabulary list. This 

2 

number is a constant throughout the present experiment. 



The actual indexing at Cranfield was not in fact constrained 
by any sort of vocabulary list. The purpose of the Cranfield II 
project was to see how retrieval performance is affected by in- 
troducing increasingly greater measures of control into the sim- 
plest form of indexing, viz . , selecting words directly from the 
natural language of the document being indexed. The purpose of 
the present experiment, however, is to test the effect on per- 
formance of varying the operational breadth of indexing; for this 
it is necessary to have a precise definition of vocabulary size. 

It is necessary, in other words, to assume some sort of initial 
indexing vocabulary or authority list. It seems the simplest way 
to do this is to regard the full set of terms used to index the 
original Cranfield II collection as constituting a hypoethetical 
vocabulary list for the 200 documents in the Salton-Cranf ield sub- 
set of the collection. 

2 

It can be observed that some of the terms on the hypotheti- 
cal vocabulary list are never actually used in indexing the 200- 
documenr. subset collection; consequently these terms have breadth 
0. It is perhaps aesthetically awkward to allow for null post- 
ings, i.e., index terms of breadth 0, and one might contrive to 
get rid of them by restricting the indexing vocabulary to those 
terms which are used at least once in indexing the 200 documents. 
On the other hand it cati be said that any definition of vocabu- 
lary size that is based on the frequency of term occurrences en- 
tails some arbitrariness in cutoff -- should one presume, for 
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The above variables are related to each other by the follow- 
ing formila: BV = DN = T^, where T is the total posting fre- 

quency. This formula prescribes the experimental method for 
varying breadth of indexing* Since V and N are constants, if the 
average depth of Indexing is decreased, i«e«, if certain index 
terms are removed from documents, this will automatically decrease 
the breadth of the terms removed, and, thus, average breadth is 
also decreased. The method of varying indexing breadth then is 

simply to vary the depth of indexing by deleting terms from docu- 
2 

ments • 



instance that breadth 0 is significantly different from breadth 
1, but that there is no significant difference between breadth 
1 and 2? There seems reason enough for defining indexing vocab- 
ulary independently of the index terms actually posted, that is, 
in terms of an assumed pre-established vocabulary list. 

(total number of postings) can be expressed in several 

ways : 

1) T = ND 

2) T = VB 

i.e., the sum over the document - 
term matrix, i ranges over the 
V vocabulary terms; j ranges 
over the N documents. 

where B (breadth) ranges over 
the z different breadths and n^ 

is the number of terms of 
breadth B* 

o 

See E. Svenonius, "The Effect of Breadth of Term on Re- 
trieval Performance," Studies in Indexing Depth and Retrieval 
Effectiveness (NSF GN 380) (Chicago; University of Chicago, 
Graduate Library School, 1968), 



V n 

3) T . ^ 
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Swanson, in his paper **On Indexing Depth and Retrieval 

Effectiveness)*^ examines the effect on precision and recall when 

the average depth of indexing is decreased in a particular way, 

viz , , by deleting index terms from documents randomly. He finds 

2 

that recall drops but precision tends to improve when terms are 
randomly deleted from documents. From only theoretical consider- 
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^Swanson, *'0n Indexing Depth and Retrieval Effectiveness,** 

2 

Precision does not always increase when terms are randomly 
deleted from documents. For instance, on the case where the re- 
trieval conditif»n is set at the highest cutoff point possible. In 
this case no documents can filter down into this overlap category 
and precision afteh: deletion is the same as it was before, Math- 
is the number of doc;uments, each with p index terms, 
which contain k index terms in common with those of 
a question q. i » 0 refers to irrelevant documents 
with respect to q; i = 1 refers to relevant docu- 
ments with respect to q. 



ematically , 



where nS , 
ipk 



d is the percentage of documents with k overlaps that 

Ipkt lost t after deletion 

and L is the ratio of relevant tc irrelevant documents 

retrieved 



n 



is indexing depth 

is number of terms deleted (See Swanson, **0n In- 
dexing Depth and Retrieval Effectiveness,**) 



Since d = d , 
IpkO OpkO 



d N 
IpkO Ipk 

n'* 

Ipk 



X 



Qpk 



= 1 



d N 
OpkO opk 



Thus L = 1 X L , there it? no change in L and consequently 
p-m p 1 

ncne in precision (precision = 1 + 1.) • 

T. 
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ations then, one can obtain at least a conditional answer to the 
question; how does the average breadth of indexing affect re- 
trieval effectiveness? The answer is that recall drops and pre- 
cision tends to improve in proportion as the average breadth of 
indexing is decreased. The condition is that Indexing breadth is 
varied by a random deletion method. Note that if consideration 
went no further, the notion of average breadth would be super- 
fluous, since average breadth is a linear function of average depth 
and anything which can be said of one of these parameters in its 
relationship to retrieval effectiveness can derivatively be said 
of the other. 

On the other hand, randomly deleting index terms from docu- 
ments is only one method of varying the average breadth of in- 
dexing, It can be asked what happens to retrieval performance 
when terms other than "the important ones" are deleted. One 
might characterize unimportant or peripheral index terms to be 
deleted as those not occurring in the title of the document, or 
as terms which are not highly weighted, or, in the case of broad 
and narrow terms, leave Judgment suspended and the question open 
to investigation. Restricting the method of varying the depth 
and breadth parameters to a random deletion process is a sim- 
plifying measure convenient as a first step in a theoretical 

1 

approach to the question. In an experimental situation, how- 

^In W,B. Rayward and E. Svenonius, "Consistency, Consensus 
Sets and Random Deletion," Studies in Indexing Depth and Retrieval 
Effectiveness (NSF GH 380) (Chicago: University of Chicago, Grad- 

uate Library School, 1967), an attempt was made to study the noi- 



68 



62 



ever, a restriction of this sort is not necessary -- it is rela- 
tively simple to study the effects on retrieval performance of 
deleting terms from documents by a systematic nonrandom method. 

This is what is done in the present experiment; broad, narrow, 
non-title terms, terms not heavily weighted, are deleted from the 
indexing of the documents. The concern is not so much with the 
effects on retrieval of varying the average breadth (equivalently: 
average depth) of indexing, as with the effects of removing par- 
ticular classes of terms, those designated as broad, narrow, etc. 
Average depth and breadth are in a sense extraneous variables; 

V 

they are therefore held constant under deletion. Whether the 

terms deleted from documents are broad or narrow, it is always the 

1 

same number of terms that are deleted. Thus, while different 
indexings result from the deletion of broad and the deletion of 
narrow terms, they represent indexings of the same average breadth, 
average depth and total posting frequency. Although the main 



random deletion of index terms theoretically. The question 
asked was what happens to retrieval performance when unimportant 
terms are randomly deleted from documents. (In the study no 
attempt was made to characterize important and unimportant terms.) 
The difficulty with such an approach is that the delition model 
becomes subject to even more simplifying assumptions, thus paling 
further its ability to depict a real situation; at the same time 
the model becomes so complex and heady it needs more to be ex- 
plained than it could explain. 

^Average breadth is diminished equally whether a broad resp . 
narrow term is deleted from a document, or whether a term is ran- 
domly deleted. This is because in each case the total number of 
postings is altered equally; V and N are constants. (If any k 
terms from the vocabulary list are deleted from documents, ave- 
rage breadth after deletion is T - k . ) 

V 
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objective of the experiment is to compare the relative useful- 
ness of broad and narrow terms, it can be said that what, in 
effect, is being compared are different nonrandom deletion meth- 
ods* From this point of view the experiment can be regarded as 
contributing to an understanding of the effect of indexing depth 
on retrieval performance in that it extends the means of varying 
depth beyond the random deletion method. 



The experiment was conveniently divided into two stages, 
corresponding to two questions the experiment was designed to 
answer: 

1) Of the index terms assigned to documents, which function 
most effectively in retrieval, the most used or popular terms, or 
those which are used relatively infrequently? 

2) Do two indexings of comparable average breadth (or 
depth) differ significantly in the retrieval results they produce 
because one consists of "quality” indexing and the other does not? 

Stage 1: In comparing the effectiveness of broad vs. narrow 

terms, terms of varying degrees of breadth were deleted from the 
documents. The first step in classifying terms as "broad” or 
"narrow" was to obtain the spread of different breadths by list- 
ing the vocabulary items in order of their frequency of occurrence, 
i.e., according to how many times they were used to index documents. 
The question was how to decide exactly which of the terms should 
be called "broad" and which "narrow." It was decided that instead 



Deletion Procedures 
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of drawing a single line between "broad" and "narrow," more 
information would be gained if varying degrees of breadth were 
allowed. The decision was to have four categories of terms: 
broad terms (those posted most frequently), relatively broad 
terms, relatively narrow terms, and narrow terms (those posted 
least frequently). These four sets or quartiles of terms were 
obtained in the followi^ manner: the total number of term post- 

ings to documents was divided by four (T/4). Then beginning at 
the top of the frequency list of vocabulary terms, i.e. beginning 
with the most frequent terms, and going down this list, the fre- 
quencies of the individual terms were summed until the number T/4 
was reached. All terms whose frequencies were summed were said to 
belong to the first quartile of terms; these consisted of the 
broadest terms used in the indexing. Continuing down the list, 
summing frequencies again equal to T/4 gave the second quartile 
of terms, and so on, for the third and fourth quartile terms. Each 
of the term quartiles thus obtained represents a subset of the 
original indexing of the documents, the first quartile consisting 
of the broadest terms, the second of the next broadest, etc. What 
these sets of terms have in common, and why they are called "quar- 
tiles," is that the total number of postings represented by the 
terms in each of the quartiles is the same, approximately T/4,^ 
or one quarter of the total term postings. A consequence is that 

^"Approximately," because it was desired that every occurrence 
of a given term be in one quartile, i.e., that terms not be "split" 
with some of their postings in one quartile and some in another; 
this required making quartiles with slightly less or slightly more 
than T/4 postings. See note 2, chap. Ill, p. 93. 
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there are many fewer unique terms in the first quartile, con- 
sisting of broad terms frequently posted, than in the fourth quar- 
tile, consisting of infrequently posted terms. 

There were four deletions of terms from documents correspond- 
ing to the four term quartiles. In the first deletion those terms 
occurring in the first quartile were deleted, the deletion of a 
term being understood to mean its removal from each of the docu- 
ments to which it was originally assigned. The retrieval operation 
was performed and the results evaluated. These first quartile 
terms were then restored to the documents from which they were 
taken, and then second quartile terms were removed. And so on. 

The breadth of an individual term determined whether or not it was 
deleted, and each of the terms used to index the documents was 
deleted in one and only one quartile deletion. At each deletion 
the same number of postings were removed from documents. The 
rationale for this was to achieve , after each deletion , indexings 
having the same average depth, average breadth and total posting 
frequency, the variable of interest being not these general sta- 
tistics but the breadth of terms deleted. (See p. 62.) 

Stage 2: In order to detiT?rmine whether title terms and 

heavily weighted terms are especially suited to be index terms, 
the retrieval power of these terms had to be tested with respect 
to a comparable set of other ^reasonable" index terms. It was 
assumed that a reasonable index term would be any of those assigned 
to a document in the original Cranfield indexing. Accordingly the 
following four deletions were made from the indexing of the docu- 
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ments : 



!♦ From each document, all index terms not also occurring 
in the title of that document were deleted* 

2* The same number of terms was randomly deleted^ from the 
indexing of that document* 

3* From each document, all index terms that were not given 
the heaviest of weightings (weight 10) were deleted* 

4# The same number of terms was randomly deleted from the 
indexing of the document* 

Deleting in this manner permitted the testing of two indexings that 
represented two subsets of the originally assigned terms, subsets 
of equal size but one consisting of "quality’* terms and the other 
of merely "reasonable" terms* 



After each of the deletions of index terms from documents , 
the collection was scanned to retrieve documents responding to the 
search requests. In the searching procedure, terms used to index 
the questions were matched with terms used to index the documents* 
Though there have been various sophisticated matching functions 
defined in the literature, the one used in the present experiment 
was a relatively simple one, viz . , only exact (machine- like) 



Note that "random indexing" is used to mean a random sample 
of the index terms that were assigned to the documents in their 
first indexing* That is, the terms are not just any haphazard 
terms, but are, rather, assumed to have some relevance to the doc- 
uments they index, in as much as they were put on the documents by 
trained indexers. In the present experiment the^i, random indexing 
is a random sample of index terms previously judged as suitable. 



Evaluation of Retrieval 
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matches were permitted. The usual definitions of "retrieved,'* 
•’precision," and "recall" were used. A document is considered 
’’retrieved" if the number of overlaps or matching document- 
question terms is equal to or greater than a certain specified 
number, called the cutoff point. Precision and recall are then 
defined with respect to the relevant documents that are retrieved, 
and precision is the proportion of retrieved documents that are 
relevant. 

The Expected Search Length measure was also used to evaluate 
retrieval performance* This measure assesses the effectiveness of 
a retrieval system in retrieving a specified number of wanted 
relevant documents (s*). The measure assumes that for each search 
question or request the collection of documents is partitioned into 
n sets or levels of documents which have, respectively, n, n-1, 
n-2 • • • terms in common with the question, the highest or top 
level of the ordering consisting of those documents which have n 
terms overlapping search request terms. The collection is searched 
for relevant documents beginning at the top level of the ordering 
and proceeding down through the lower levels until the requisite 
number of relevant documents is found. The "Expected Search Length" 
(esl) is a measure, a statistical expectation, of the number of 
irrelevant documents that must be gone through before the request 
is satisfied. Thus esl values increase as the system performs 
less effectively. The formal definition of esl is as follows: 

^The choice of matching function is somewhat dependent on 
how "index term" is defined. See footnote on p. 79. 

o 
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esl (q) = j + is 
r + 1 

where q is the request for which esl is being determined 

j = the number of irrelevant documents found at levels 
preceding the level where the request is satisfied 

i = the number of irrelevant documents at this level 

r = the number of relevant documents at this level 

s = the number of relevant documents at this level 

required to satisfy the request 

There is a rough correspondence between esl and the precision- 
recall measure. In a search through a collection of documents par- 
titioned according to overlap, a request for a single document 
(s'* = 1) is generally satisfied at a high overlap level. Thus 
setting the retrieval condition at a high cutoff level can be 

regarded as analogous to making a request for a single or few doc- 

uments. On the other hand, if many documents are requested of the 
system (for instance all relevant documents), 1l is not likely 
that the request can be satisfied at a high overlap level. In 
fact, if all relevant documents were wanted and it happened that 
one of these had no terms at all in common with the question, the 
search for this document would have to proceed down from the high- 
est overlap level through all the succeeding levels until the 
document was eventually found at the 0 overlap level. Often set- 
ting the retrieval condition at a low cutoff level is analogous, 
to making a request for many documents. A low s* value can be in- 
terpreted to mean the user is more interested in precision than 
recall, a high s* value can mean recall is preferred to precision. 
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The Statistical Test 



The Wllcoxen matched -pairs signed-ranks test was used to 
determine whether the esl, precision and recall values obtained 
after the deletion of terms were significantly different from the 
corresponding values before deletion. Also compared for signifi- 
cance were different indexings of the same depth: indexing with 

title terms vs. indexing with the same number of terms chosen ran- 
domly from the original indexing; indexing with heavily weighted 
terms vs. a random selection of the original terms to a like depth. 

The Wilcoxen Test is a nonparametric test. That is, it is not 
conditional upon assumptions regarding the nature of the population 
from which the sample is drawn -- it need not be assumed, for 
example, that the sample values are drawn from a normally distrib- 
uted population. There are advantages and disadvantages to non- 
parametric tests, the main advantage being, of course, that they 
are "distribution-free. ” The relaxing of conditions contributes 
to the main disadvantage of the tests, viz . , they are not as power- 
ful as the corresponding parametric tests when the conditions 
needed for the parametric model are satisfied. In the present 
experiment making any assumptions about the population distribution 
would be unrealistic and there was really no option in deciding be- 
tween a parametric or a nonparametric model; the nonparametric one 
had to be used. Since the differences that were being judged for 
significance were between two related samples (eg., the precision 

^For a more extended description of the test than is given 
here see Sidney Siegel, Nonparametric Statistics for the Behavioral 
Sciences (New York: McGraw-Hill, 1956). 
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values yielded by two different indexing methods), and since the 
values being compared represented ordinal measurements, either 
of two nonparametric tests could have been used: the sign test or 

the Wilcoxen Test. The tests are rather similar and either would 
indicate whether one indexing method is significantly better than 
another. The Wilcoxen Test, however, has an advantage over the 
sign test in that it can be used to tell not only that one index- 
ing method is better than another, but also how much better. 
Exploiting information about the magnitude as well as the direc- 
tion of the observed differences, the Wilcoxen Test is thus the 
more powerful test, and the one that was chosen for the experiment. 

The test makes use of the null hypothesis, the hypothesis of 

no differences. It is hypothesized that two methods of indexing 
are equally good; for instance , indexing with a subset of the index 
terms originally assigned to a document, such as those terms 
which also occur in the title of the document, gives as good pre- 
cision values as indexing with the full set of assigned terms. 

In other words, it is assumed that for any given search question, 
chance alone determines which of the indexings gives the best 
retrieval results. The null hypothesis is stated here, as is 

usual, in the hope that it can be rejected and that the alternative 

hypothesis, viz . » that there is a significant difference between 
the two indexing methods, can be accepted. To reject the null 
hypothesis the observed differences in the sample comparisons must 
be such that they disagree **enough** with the expectation of no 
difference. That is, the probabilities associated with these ob- 
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served values must be small, in particular less than or equal 
to a critical value Ct * In the present experiment the data were 
analyzed for critical values of .05 and .01. As no assumption was 
made as to the direction of the differences, a two-tailed test 
was used. 

The samples being compared in the experiment consisted of 
sets of retrieval values associated with the individual search 
questions. For instance, the recall values for each of the 42 
questions before deletion were matched with the corresponding 
recall values after deletion* Each matched pair was given a dif- 
ference score dt • The next step was to rank the s for the 42 
questions without respect to sign and omitting tied scores. Each 
rank was then given the sign of the difference it represented, 
tied d's being given the average of the tied ranks. The rationale 
of the test is that if the null hypothesis were to hold, the sum 
of the ranks having a plus sign would be about the same as the 
sum of the ranks having a minus sign. The null hypothesis is 
rejected if either of the sums is too small. Significance was 
determined first by finding the smaller of the sums of the like- 
signed ranks, T« Then, if the sample size was less than 25, a 
table of critical values for different significance levels was 
consulted to see if T was small enough to warrant a judgment of 
significant difference. If the sample size was larger than 25, 
then £, the probability of the observed difference, was determined 
by computing: 
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T - N (N + 1) 

4 

^ N(N + 1) (2N + iT 

^ measures the closeness of T to the mean of summed ranks, and a 
table gives the probability £ associated with the value of z at 
a given significance level. The null hypothesis is rejected if 
£ is less than or equal to Qf . 

A comment about the use of the statistical test in this 
experiment: although there are 42 questions, the samples compared 

did not always consist of 42 values. Partly, of course, samples 
were depleted by the occurrence of tied scores* In the case of 
precision, however, a more serious reduction in sample size 
occurred owing to the fact that: for many questions precision val- 
ues were indeterminate at the higher cutoff values since nothing 
was retrieved and, formally, precision equaled 0/0; and these 
questions consequently had to be omitted from consideration. 

This was unfortunate, since the more the sample size is diminished 
the less power the statistical test has and the more limited the 
generalizability of the results. 

Even in a more general way the inferences which can be drawn 
from the experimental data, using statistical methods, are lim- 
ited -- they are limited by the context of this single and rather 
small experiment. It is not clear, and it cannot be presumed, 
that the sample data used in the experiment is representative of 
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a population consisting of any search request addressed to any 
collection of documents* It is possible that the results are de- 
scriptive only of this particular collection and this particular 
set of questions* 



The Dat a 



The original Cranfield II collection consists of 1400 docu- 
ments and 279 search questions* Most of the documents are on the 
subject of aerodynamics, but a small number of them deal with air- 
craft structures*^ The effect of the subject area of the collecticn 

2 

on the test results is an unknown and as such qualifies the gen- 
eralizability of any results using the full data or a subset of 
the data* The 1400 documents comprise 173 "base documents," 1018 
papers that were cited in the base documents and 209 "further" doc- 
uments selected by students or by the method of bibliographic 
coupling for their presumed relevance to the set of search 



The purpose of this was to examine the effect of two dis- 
similar subjects included in one collection* See Cleverdon, Mills 
and Keen, Factors Determining the Performance of Indexing Systems , 
Vol* 1, 19* 

2 

"We find it impossible to say categorically that the subject 
area of the test collection did not have an influence on the * * * 
results*" (Cleverdon and Keen, Factors Determining the Performance 
of Indexing Systems ^ Vol* 2, 256*) However, Salton points out that 
the Cranfield documents are substantially more technical in nature 
and the collection as a whole more homogeneous than most* He spec- 
ulates that because of the homogeneity of the collection word nor- 
malization and synonym procedures are not so effective as for 
other collections* (Salton, "Computer Evaluation of Indexing and 
Text Processing," p* 28*) 
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questions . ^ 

The 279 questions were obtained from the authors of the base 
documents* Each author was asked to state in the form of a search 
question the reason he had for undertaking the research that led 
to the writing of his paper* He was asked also to compose up to 
three supplementary questions which either arose in the course of 
his work or which could be imagined as being put to an information 
service* A total of 640 questions was obtained, but of these only 
279 were selected for use in the experiments* To be selected a 
question had to satisfy the criteria: it had to be grammatically 

complete; and there had to be two or more documents in the collec- 
tion assessed as relevant to the question by the author submitting 
the question* 

Each document was examined for its relevance to each question, 
an impressive and "onerous" task involving over half a million 
judgments* The procedure was as follows: each author, when asked 

to formulate search questions on the basis of his paper, was at 
the same time asked to judge the documents he cited in his paper as 
to their relevance to the questions* Relevance was judged accord- 
ing to a 5-degree scale, from relevance 1 for documents which 
completely answered the question, to relevance 5 for documents 
of no interest whatsoever* There were, of course, in the 1400- 

^While not available for the full document collection, certain 
characteristics of the base documents are given: each paper con- 

tained in its bibliography at least two references in English that 
were dated 1954 or later: over half the papers were from one 

journal, the Journal of the Aerospace Sciences; for the irost part 
the authors and bibliographic origin of the papers was American* 
(Cleverdon, Mills and Keen, Factors Detenolning the Performance of 
Indexing Systems * Vol* 1, 19, 20*) 
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document collection, documents other than those cited by the author 
that were relevant to his questions -- so every document in the 
collection had to be examined in relation to every question. The 
task of "assessing relevance" was performed preliminarily by five 
students in the field of aerodynamics. But also, as a supplemen- 
tary measure, preliminary assessments were obtained "automati- 
cally" by bibliographic coupling: a document was considered rele- 

vant to a search question if it had 7 citations in common with 
the base document that generated the question. The last and cru- 
cial step, i.e., the actual relevance judging, was to send the 
abstracts of documents thought relevant to a particular question 
to the author of the question, asking him to make the decision 
whether the relevance assessments obtained by the students and by 
bibliographic coupling were actually valid. Asking the author of 
a question to be the final arbiter of relevance is realistic , and 
the objection cannot be made that there were documents judged 
relevant that were really irrelevant. It is not clear, however, 
that all documents relevant to the questions were found — that 
is, some documents preliminarily assessed as irrelevant might 
really have been relevant. This is another factor that must be 
considered as qualifying the results obtained using this data. 

(A fuller discussion is given in the next chapter, pp. 121 ff ) 

On the average seven relevant documents per question were found. 

The indexing of the documents in the collection was a post- 
coordinate natural language indexing. Natural language phrases, 
consisting of one or more words, were assigned to the documents. 



82 ' 



76 



These phrases presumably designated "concepts." The terminology 

is perhaps misleading or unnecessary^ — the condition imposed on 

the indexing by the use of the word "concept" is that vague and 

ambiguous words, words which taken singly might be useless as 

retrieval handles (eg., modifiers such as "high"), had to appear 

in conjunction with other words which would provide context and 

thus, perhaps, a more solid image. Each of the concepts was 

weighted by assigning to it a value indicative of its relative 

importance in the document. A range of six weights was adopted, 

the highest weight being 10 and the lowest 5. For the most part 

weights were assigned on a subjective basis, depending on whether 

the concept was in the main general theme of the document or in 

a major or minor subsidieary theme. Individual terms within a 

concept were given the weight of the concept, and if a term 

appeared in more than one concept in a document, it was given the 

2 

weighting of the more heavily weighted concept. 

The main objective of the Cranfield effort was to test the 

effect of various index languages on the performance of retrieval 

systems. Indeed this is the rationalization for the hypothesis 

of concepts --a concept language was one of the languages to be 

tested. Another index language tested at Cranfield was called a 

2 

"single term language." The name is somewhat wrong in that a 
single term can be thought of as consisting of one or more words, 

^See footnote 1 on p. 56. 

^Cleverdon, Mills and Keen, Factors Determining the Perform - 
ance of Indexing Systems , Vol. 1, 55. 

^Ibid., p. 59 ff. 

O 
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but what is Intended is that index terms be regarded as single 
words or descriptors. This language was formed by taking individ- 
ual words out of the context of the phrases in which they appeared. 
For instance, if "finite wings" was assigned to a document, this was 
to be regarded as equivalent to assigning the two terms "finite" 
and "wings." At Cranfield a vocabulary list of the unique single 
word terms used in the indexing of the 1400 documents was drawn 
up. In constructing the list, use was made of certain minor nor- 
malizations ^representing "initial controls" of the indexing lan- 
guage* Since roughly the same procedure was followed in the present 
experiment, these normalizations or initial controls are listed: 

"(1) Singular and plural forms were confounded. 

"(2) American and English and other variant spellings were 
confounded: e.g., gage and gauge, fiber and fibre, Von Karman and 

Karman. 

"(3) Certain qualifiers of terms (affixes, hyphenated forms 
which were sometimes separated, etc.) were disregarded, e*g., 
builtup, pitch-up, rolled-up, etc. were treated as built, pitch, 
rolled; ellipse-like, jetlike, etc., were treated as ellipse, jet. 

"(4) Numbers as qualifiers were separated and treated as 
separate terms; e.g., Mach 6 became *Mach* and *6*, N.P.L. 18 x 4 
(a wind tunnel) became *N.P.L. * and *18 x 4*.** The vocabulary 
list numbered 3094 single words. In the indexing of the 1400 
documents, the total number of "single term** postings was 43,857, 

^Cleverdon, Mills and Keen, Factors Determining the Perfor - 
mance of Indexing Systems . Vol. 1, 58. 
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the average depth of indexing was 31.3 terms per document and the 
average indexing breadth was 14,2. 

What has been described are some of the basic features of 
the Cranfield II experimental data. In the present experiment it 
was a subset of this data that was used, a subset consisting of 200 
documents and 42 questions. The documents and questions were 
chosen, with the help of a member of the Cranfield group, by 
Professor Salton for his experimenting at Harvard with the SMART 
program, A characteristic of this smaller subset of data is that 
all documents in the 1400-document collection that were relevant 
to one of the 42 questions were included in the 200 -document sub- 
set.^ There were on the average 4,7 relevant documents per question. 
A possible objection to the Salton-Cranfield data is that there 

are too few documents and questions to yield results that are gen- 

2 

eralizable. It is believed, however, at least at Cranfield, that 
the smaller set of documents is representative of the larger data 
base, especially for the case of single term indexing. That is, 
already in previous experimenting the data has been accepted as 
a reliable inferential base. While this is no guarantee, the 



This is not strictly true. Document 1329 is relevant to 
question 119 and is not included; nor is document 2289 which is 
relevant to questions 145 and 146. 

2 

"Undoubtedly the size of the test collection , , , is 
smaller than one would have liked. The test results presented in 
Chapter 4, Section 1, show that the smaller sets of documents and 
questions were representative of the complete document collection 
and question set, but these tests were only concerned with the 
Single Term index languages « , .However, there appears to be no 
justification for suggesting that the size of the test collection 
could have significantly affected the comparison between systems." 
(Cleverdon and Keen, Factors Determining the Performance of In - 
dexing Systems . Vol. 1, 256. 
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availability of the data, the fact that the documents are more 
deeply indexed than in any other document collection (and by pro- 
fessional indexers) and the integrity of at least those relevance 
judgments made contribute to making the data the best possible for 
the present experiment. 

The data tape sent by Mr. Salton contained the weighted con- 
cept or phrase language indexing of the documents, a listing of 
the titles and abstracts of the documents, the questions and a 
record of the relevance assessments. In the present experiment 
use was made of only the indexing of the documents. Questions, 
relevance assessments and document titles were taken from the orig- 
inal Cranfield publications. To prepare the data for use, before 
the programming could begin, the following tasks had to be per- 
formed. 

1. Convert the phrase indexing of the documents to a single 
term indexing in accordance with the normalization procedures fol- 
lowed at Cranfield. 

2. Index the questions. 

3. Index the document titles. 

4. Construct a relevance file, associating with each question 
the documents relevant to it. 

Though something may be lost in the breaking up of phrases, 

there were definite reasons in the present study for preferring 

1 

index terms to be single words rather than phrases. And, for the 

^One advantage to a single word index language is that it is 
familiar -- keywords and descriptors being now part of standard 
index usage. The advantage is not negligible in that, in the pasti 
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results of retrieval experiments have tended to be somewhat blurred 
by an overelaborateness and consequent confusion in the details 
of the experiments# Where possible it seems advisable to let defi- 
nitions conform as much as possible to what is easily and already 
understood • 

A second reason for preferring index terms restricted to sin- 
gle words is to safeguard against the possibility that certain 
deletion procedures could be made to look problematic* For instance, 
suppose **wings** is deleted from one document and "finite wings" 
from another# It can be argued that while the same number of 
"terms" are being deleted, there is an important difference in the 
units of meaning being removed. The argument would be difficult 
to meet if one concurred in the view that a word constitutes a unit 
of meaning. It is, of course, claimed both ways, that as units of 
meaning words are too small (context must be considered) and they 
are too big (the unit of meaning must be found at the morphological 
level). Both claims seem reasonable and rather than getting 
involved, it seems safest to con^romise with common sense by con- 
fining index terms along with units of meaning to word boundaries# 

The only problem then is what is a word boundary, see p. 81# 

The use of single words as index terms has the further advan- 
tage of permitting retrieval to be carried out on the basis of a 
relatively simple matching function. The matching function is 
used to determine the number of overlaps between a document and a 
search question. For instance an exact (machine- like) match of 
a question word with an index term assigned to a document consti- 
tutes an overlap ( wings "matches" wings ) . Admitting word phrases 
as index terms, however, introduces cases which are not so clear. 
Does wings match finite wings ? Should this be counted as an over- 
lap J If not, some degree of similarity between document and 
question is missed. If, on the other hand, it is admitted as a 
partial (generic) overlap then immediately another variable is 
introduced into the experiment, viz., search strategy. This is 
not to deny the importance of the search strategy variable, only it 
seems advisable to suppress this complication where the emphasis 
of the experiment is on other factors. 

Finally, a reason for preferring single-word index terms — 
one which does not so much rest on arguments from simplicity of 
design -- is that the use of single words as retrieval hooks does 
in fact adequately serve the purpose of retrieval. In discussing 
the results of the Cranfield II project, Cleverdon writes: "Quite 
the most astonishing and inexplicable conclusion that arises from 
the project is that the single term index languages are superior to 
any other type." (Cleverdon and Keen, Factors Determining the Per - 
formance of Indexing Systems > Vol. 2, 252.) This is all the more 
impressive as 33 different index languages were compared in the 
Cranfield project. 
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most part, it seemed convenient to adopt the word normalizations^ 
suggested by Cleverdon (see p# 77) • There was one exception: 
numbers were not used as retrieval terms# Curious to observe is 
that neither are numbers included in the Cranfield vocabulary list 
of terras used in the indexing of the original 1400 documents -- 
the reason, perhaps, being that numbers by themselves, in isolation, 
are not especially descriptive and might thus be regarded as too 
common to be used as retrieval terms • In any case, it seems a 
matter of fairly arbitrary decision whether or not numbers are used 
in the retrieving of documents. The first task, converting the 
phrase indexing of the dociiments to single term indexing, involved 
another more problematic decision. The trouble lay in the defi- 
nition of a word. On the Cranfield vocabulary list sometimes a 
combination of letters would appear as a single word, but in the 
actual indexing of the documents these letters would appear as two 



Word normalization procedures are a matter of degree: sin- 

gular and plural forms of one word might be regarded as the same 
term; further, word variants, such as **fix** and "fixingV might be 
regarded as equivalent; still further, synonyms might be com- 
pounded and regarded as identical. The literature shows some dis- 
crepancies in judging the usefulness in retrieval or word normal- 
ization procedures. Cleverdon finds that grouping true synonyms 
and word forms makes for improvement in performance, but the im- 
provement is relatively small (Cleverdon and Keen, Factors Deter - 
mining the Performance of Indexing Systems # Vol. 2, 253.) Salton 
finds that the more thorough normalization inherent in matching 
word-stems improves search effectiveness less than compounding 
singulars and plurals, Salton, however, agrees with Cleverdon in 
admitting that the differences one way or another are not dramatic. 
(Salton, ''Computer Evaluation of Indexing and Text Processing," p. 
28,) In light of the nonconclusiveness , it seems as well to 
adopt the minor normalizations used in constructing the initial 
Cranfield vocabulary list of 3094 single terms. 
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words. For instance, **lessthan” is a single word in the vocab- 
ulary listing of terms, but on documents it is written as two 
words. Conversely, sometimes two words on the vocabulary list are 
run together in the indexing of the documents and appear as one 
word, e.g., ”navierstokes" and **navier stokes." Something had to 
be decided and the decision was to let the indexing of the docu- 
ments be determining in the following way: unbroken sequences of 

letters that occurred in the indexing phrases assigned to documents 
should be included in the vocabulary list as "single words," except 
if these were obviously common words, common being decided on by 
the Cranfield policy for prepositions, adverbs, etc., which is 
inferable from the Cranfield vocabulary list. In this manner 
revisions were made and a modified vocabulary list was drawn up 
which consisted of 3168 terms, as compared with the Cranfield list 
of 3094 terms. The Cranfield list is given on pp. 221 - 236 of 
Clever don, Mills and Keen, Factors Determining the Performance of 
Indexing Systems . Vol. 1. The revised list used in the present 
experiment is given in Appendix A. 

Cards were then keypunched for each document-term pair, 
showing also the weighting of the term with respect to the docu- 
ment.^ The cards were of the following form: 

Document Number Weight Index Term 

1302 05 BOUNDARY 

^Some of the weights were missing from terms on the Salton 
tape. In some cases they were dispensable, there being other 
occurrences of the same term on the same document. Otherwise the 
weight was regarded as indeterminate. 
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The cards were alphabetized and duplicates were discarded* A 
listing of these cards » showing the index terms assigned to the 
200 documents, is given in Appendix B. 

The indexing of the questions and document titles was kept as 
automatic (and repeatable) as possible* The rule followed: a 

word occurring in a question or a document title was to be consid- 
ered an index term if, under minor normalization (p. 77), it 
matched a word on the revised vocabulary list* The questions, with 
the index terms assigned to Ihem, are given in Appendix C. The 
cards in the question-term file were of the form: 

Question Number Index Term 
100 BOUNDARY 

The indexing of the document titles can be inferred by looking at 
the listing of the records in the document-term file* A T was 
added to a record if the index term assigned to the document was 
represented in the title of the document (it is only title terms 
satisfying this condition that are of interest in the experiment, 
see p* 65)* In making the relevance file, the Cranfield distinc- 
tion of different relevance categories was not maintained, docu- 
ments of relevance 1 through relevance 4 were considered simply as 
"relevant." The cards in the question-relevance file took the 
form: 

Question Number Number of Relevant Relevant Documents 

Documents 

100 4 1785 1786 1787 1788 

A listing of the cards is given in Appendix D. 



ERIC 



93 



84 



Programming the Experiment 

The data files used in the experiment are the document-term 
file, the question-term file and the question-relevance file. 
Before beginning the actual deletion and retrieval operations, 
three preliminary tasks, for which programs were written, were 
performed. 

1. The terms used in the indexing of the documents were 
ordered according to their breadth or frequency, and the depth, 
breadth and posting statistics were taken. The total number of 
postings was divided by four, and the terms accounting for the 
postings in each quartile were separated into four not quite but 
almost equal sets.^ For use as deletion decks, the quartile term:> 
together with their frequency of occurrence were output on punched 
cards. A list of the full set of terms divided into the four sub- 
sets is given in Appendix E, and the distribution graph is shown 
in Appendix G, p. 93. 

2, The number of non- title and non-weighted-10 terms in each 
document was counted -- the information needed for this being 
already punched on the records in the document- term file. Sepa- 
rately for each document and for each of the two types of terms, 
the same number of terms was selected at random from the original 
sei of index terms assigned to the document. The program to do 
this punched on a V on a record in the document-term file if the 
term was selected as a member of the randomly chosen set corre- 

^See footnote 2, p. 412, Chapter III. 
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spending to non-weighted-10 terms; an S was punched if the term 
belonged to the random subset corresponding to non-title terms* 

Thus in the document-term records for a given document, the records 
marked with S equaled in number those not marked with a weight of 
10 . 

3. From the document-term file all records for index terms 
that were not also used in the indexing of the questions were 
removed* This reduced -he size of the file necessary to search 
through in the retrieving operation by about a third. The reduc€nd 
file was called the small document file and the original file the 
large document file* A listing of the small document file is 
given in Appendix F* Note the terms are alphabetically ordered 
and following each term are the numbers of the documents that ar^ 
indexed by the term* The reason for this format was to enable one 
to manually simulate (and check) the automatic retrieval operation 
by matching question terms (Appendix C) with the document terms in 
the small file (Appendix F) * 

The deletion operations were as follows* The four deletion 
dwcks showing the index terms together with their frequency of 
occurrence were each alphabetically ordered* The terms in each deck 
were then matched against those on the records in the small 
document- term file, which also was alphabetically ordered* A 
match signaled that the document-term record was to be deleted* 

(The listing in Appendix E showing the frequency distribution of 
terms is divided into four parts, indicating, thus, which terms 
were removed from the document indexing in each of the quartile 
deletions). The deletion of non-title terms, non-weighted-10 
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terms, V and S terms, did not require the matching of two sets 
of records « A program was written to delete a term-document 
record if it did not bear a T; if it did not bear a weight of 10; 
if it did have on it a V; if it had on it an S. (Which terms were 
removed in each of the deletions can be seen by referring to the 
document-term records listed in Appendix B.) After each deletion, 
depth, breadth and posting statistics were taken. 

In the retrieval operation, records in the question-term file 
were matched against records in the small document file, each file 
being alphabetically ordered. Only exact, machine-like matches 
were permitted. Thus 100 BOUNDARY, a record in the question-term 
file, had a match with 1302 15 BOUNDARY, a record in the 

document-term file. For each question all documents retrieved by 
the individual vjords in the question were ordered according to fre- 
quency of occurrence. For instance, one document might respond to 
five words in the same question. This document then, is said to 
have a frequency of 5 for that question -- which is simply another 
way of saying that the document is retrieved in response to the 
question at the 5 overlap level. The retrieval operation just 
described can be replicated by matching the question terms in 
Appendix C with the document terms in Appendix F* Fig. 1 is a 
sample of what the ordered retrieval output for one question would 
have looked like at this point, had it been printed out. 

The next step was to record which of the retrieved documents 
were relevant. This was accomplished by consulting the question- 
relevance file in which documents judged relevant to each question 
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were listed. In the set of ordered documents retrieved by a 
given question, a document relevant to the question was marked 
with a star (see Fig. 1). 

The retrieval output for each question was then evaluated. 
Recall and precision were determined at 5 overlap levels (cutoff 
points), k, denoting cutoff value, ranged from 1 to 5.^ The me- 
chanics of the evaluation consisted simply of plugging in the appro- 
priate formulas the values obtained by counting different sets of 
documents in the ordered retrieval output. For each question q 
and for each cutoff point k. 



number of starred documents retrieved 

Precision of frequency ^ k 

total number of documents retrieved 
of frequency k 



number of starred documents retrieved 

Recall of frequency ^k 

total number of documents relevant 
to question q 



(The program was directed to the question-relevance file to get 
the value for the denominator of the recall formula, i.e., the 
total number of documents relevant to question q,) 

Expected Search Length was determined for five values of s* 
ranging from 1 to 5. The esl values were obtained by evaluating 



^Larger values of k were not considered for the reason that 
the average number of documents relevant to any question was 4.7, 
making it improbable that for cutoff points higher than 5 enough 
documents would be retrieved to make the results statistically 
meaningful. Similarly, values of s* above 5 were not considered, 
it being unlikely that requests for more than 5 documents could 
be satisfied in sufficient number to make the results statisti- 
cally worthwhile. 
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the esl formula for each ordered document set. Let k = the fre- 
quency or overlap level at which the request is satisfied* 



i = number of unstarred documents of frequency k 
s = s* - X (s* is the number of documents requested) 

There were some special cases in the esl program. It was neces- 
sary before using the esl formula to ascertain whether or not the 
request being considered could be satisfied in the document col- 
lection. For instance, it had to be asked if there indeed existed 
five documents relevant to the search question. If the answer was 
yes, the formula was applied; if not, the question was given no 
esl value and was not included in the averages. Another problem 
was that it sometimes happened that a request could be satisfied 
but only at the 0 overlap level. In the retrieving operation, doc- 
uments were ranked according to a frequency of 1 or greater, that 
is, only documents with at least 1 overlap in common with a given 
question were "retrieved." Unretrieved documents, documents with 
0 overlaps in common with the question were ignored, i.e., skipped 
over, by the program. This, while efficient for the most part, had 
the difficulty that for those cases where k = 0, the esl formula 
as it appears above would not work. Thus for the case where k * 0: 
when s* is not satisfied by the time the end of the first overlap 
level is reached, and there are still relevant documents in the 



esl (q) = j + si 

r + 1 



where j = number of unstarred documents of frequency 



X = number of starred documents of frequency 




r = number of starred documents of frequency k 
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collection; a subprogram was introduced, by which the esl var- 
iables were differently defined. 

j = number of unstarred documents with frequency ^ 1 
X = number of starred documents with frequency ^ 1 
y = number of documents relevant to the question in hand 
r = y - X 

i = 200 - (x + j) - r 
s = s* - X 

Printed output was obtained at this stage. It took several 
formats, one format being a 42 x 15 matrix, the columns precision, 
recall, and esl values, the rows question numbers: 



Question 


Cutoff 1 


s*=l 


Cutoff 2 


s*=2 . 


. . Cutoff 5 


s*=5 




P 


R 


esl 


p 


R 


esl 


P R 


es 1 


7079 


0.05 


1.00 


22.00 


0.14 


0.33 


38.00 . 


. . 0.00 


— 


7100 


0.02 

• 


1.00 


1.50 


0.04 


0.75 


18.31 . 

. . 


. . 0,00 

4mm m 


• 


7360 


• 

0.05 


1.00 


2.18 


0.12 


1.00 


. . 

2.37 . 


4mm #1 

. . 0.00 


• 



The output in matrix form was punched on cards, one row to a 
question, for the purpose of facilitating two further operations: 
obtaining the mean and standard deviation values for precision, 
recall and esl, and running the test to determine statistical sig- 
nificance. The mean and standard deviations were found for each 
column of values. In the case of precision and esl, “dummy** values 
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signifying 'Value indeterminate” or "request unsatisfied” were 
excluded from the averages. Then each column of values (e.g., 
precision values for each of the 42 questions) obtained by retriev- 
ing documents after each of the deletions was compared for signifi- 
cant difference, using the Wilcoxen Test, with the corresponding 
column of values for the no-deletion case. Corresponding columns 
of values were also compared for the non- title vs. S deletion case 
and for the non-weighted-10 the V deletion. Output of the 
statistical program showed the ranking of differences in values as 
well as values for z, and ()^ (see Wilcoxen Test description, 

pp. 69 - 73). 



Summary 

The experiment described consisted of two stages. In both 
stages the indexing of a collection of documents was varied by the 
systematic deletion of index terms from documents. Documents were 
retrieved in response to search questions after each of the 
deletions and the output was evaluated in terms of precision, re- 
call, and Expected Search Length. The Wilcoxen Test was used to 
deteirmine whether the difference in retrieval effectiveness in the 
before-and-after-indexings was statistically significant. 

In the first stage of the experiment index terms were non- 
randomly deleted for the purpose of testing the effect on retrieval 
performance of varying the specificity or operational breadth of 
the indexing. In four separate deletions index terms of four dif- 
ferent breadth categories were deleted from the indexing of the 
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documents* The question asked was, "Of the index terms assigned 
to documents, which function most effectively in retrieval, the 
most used or popular terms, or those which are used relatively 
infrequently?” 

The second stage of the experiment was designed to meet the 
objection that if there is something which distinguishes good 
indexing from bad, then this is to be found in the relationships 
individual terms bear to the documents they index, in their tex- 
tual warrantability rather than in a statistical property such as 
operational breadth* Index terms assigned to a document but not 
represented in the title of the document were deleted and the 
indexing was compared with a set of terms chosen randomly from the 
assigned terms to a comparable depth* Non-weighted-10 terms were 
deleted, terms were randomly deleted to the same depth and the 
two indexings were compared for retrieval effectiveness* The 
question asked in the second stage of the experiment was, "Do two 
indexings of comparable average breadth (depth) differ significantly 
in the retrieval results they produce because one consists of *qual- 
ity* indexing and the other does not?" 
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CHAPTER III 



RESULTS 



Indexing With Broad and Narrow Terms 



Depths Breadth and Posting Statistics 

In the first stage of the experiment both broad and narrow 
terms resp , were deleted from documents and the retrieval effec- 
tiveness of the indexings before and after deletion were compared. 
Figure 2 (p. 412) shows the distribution, before deletion, of the 
terms used to index the documents in the collection,^ The four 
horizontal lines cutting the distribution curve indicate the 
upper and lower bounds on posting frequencies of the four sets of 
terms corresponding to the four separc^te deletions. In each 
deletion terms comprising twenty-five percent of the total number 
of postings were removed from documents in the collection. In the 
first deletion broad terms were removed, those occurring between 
32 and 143 times (first quartile deletion). Secondly, terms with 



^Note that the distribution is plotted on log-normal graph 
paper. The vertical axis (log-scale) shows x = number of postings, 
the horizontal axis (normal probability scale) shows S = the per- 
centage of terms having x or fewer postings. Quite predictably, 
the distribution of index term usage is representable as a straight 
line, that is, the distribution is log-normal. See Nona Houston 
and Eugene Wall, “The Distribution of Term Usage in Manipulative 
Indexes,” American Documentation , XV, (April 1964), 105, 
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13 to 32 postings per term were deleted (second quartile deletion) 

In the third deletion terms with a frequency from 5 to 13 were 

deleted (third quartile deletion). Finally, narrow terms^ were 

deleted 9 those posted between 1 and 5 times* With each deletion 

2 

the number of postings removed was 5174, thus at each deletion 
average depth was decreased from 34.5 to 25.9 and average breadth 
from 2,17 to 1,63, Table 1 shows the depth, breadth and posting 
statistics for the indexing of the collection before any deletion 
and for the four indexings resulting from the original indexings 
altered by deletion. 



Recall Statistics 

Recall results are shown in Tables 2,4 and 5 and in Figure 
3 (Appendix G) . Table 2 shows the mean recall values over the 
questions in the sample for each quartile deletion, at each of 
five cutoff points. Next to each value is written the standard 
error. The standard error in the recall value is computed from 
the standard deviation of the sample measurements by dividing the 
standard deviation of the sample, s, by the square root of N: 



^Vocabulary terms which were not used in the indexing of the 
200-document collection were not "deleted,” Strictly then the 
narrowest terms were never deleted. In the discussion of the 
results the expression "narrow terms" refers to terms posted be- 
tween 1 and 5 times, "Relatively narrow terms" are terms posted 
between 5 and 13 times; "relatively broad terms" are those with 
13 to 32 postings per term; and "broad" terms are those with 32 
to 143 postings. 

2 

5174 is an average, the range of values is from 5170 to 5178 
It was desired that every occurrence on a document of a given quar 
tile term be deleted, which made it impossible to delete exactly 
the same number of postings each time* See p* 64, 
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O'- = S . 




It can be seen from Table 2 that the standard error in the 
recall values is consistently fairly small, ranging from 0*01 to 
0.06. An error in this range is of small importance at low cut- 
off points where the recall figures tend to be high. At high cut- 
off points, hwever, the error becomes proportionately quite large. 
Generally, as will be seen, any values obtained at high cutoff 
points are less reliable than those obtained at low cutoff points, 
and must be regarded with a certain suspicion. 

Table 4 shows the recall values for the no-deletion case and 
for each quartile deletion, and these are juxtaposed with the cor- 
responding precision values. For both recall and precision, it 
is indicated when a particular value differs significantly from 
the corresponding value in the no-deletion case. 

Figure 3 shows the precision- recall operating curves for 
indexing before deletion and the indexings after each of the quar- 
tile deletions. The operating curve nearest the top rightmost 
corner represents the best retrieval: 100% precision and 100% 

recall. Correspondingly the curve falling closest to the bottom 
left of the graph represents the poorest retrieval. 

Table 5 should be looked at in conjunction with Fig. 3 show- 
ing the operating curves, in that it indicates when distances 
between an> two of the curves, e.g., indexing with 1st quartile 
terms removed and indexing with 4th quartile terms removed, differ 
significantly. 



O 

ERIC 



101 



95 



Recall Drops in Proportion to Breadth of Term Deleted 

As was to be expected, recall generally drops as index terms 
are deleted from documents. The largest drop in recall comes with 
the deletion of the broadest terms and the smallest with the 
deletion of the narrowest terms. The fact that recall drops in 
proportion to the breadth of term deleted is not entirely obvious. 
It is not obvious since at each deletion the same number of post- 
ings were deleted, that is, the same number of retrieval hooks 
were removed from documents, independently of whether the terms 
deleted were themselves broad or narrow. But then it can be ob- 
served that the factor operative here is not postings but overlaps. 
Confining attention for the moment to the number of documents 
retrieved (rather than to recall per se), it seems clear that the 
amount of material retrieved is determined by the specification of 
the retrieval condition, viz*, by the specification of the number 
of index terms a document and question must share for the document 
to be considered retrieved in response to the question. The num- 
ber of overlaps that can be associated with any term is the fre- 
quency of the term in the question indexing times its posting fre- 
quency in the document collection. In that terms frequently 
posted on documents tend also to be used frequently to index 
questions, it is more probable that a broad term will account for 
an overlap rather than a narrow one. Thus it is more likely that 
a greater number of documents will be lost from a given overlap 
category when broad, as opposed to narrow, terms are deleted. In 
other words, it is not the number of postings alone, but also the 
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number of times different terms are used to index questions that 
determines the number of documents retrieved by the terms. ^ It 
is perhaps obvious that the amount of material retrieved in a 
system is not a simple function of the total number of terms posted 
to documents in the collection. This fact, however, is useful in 
explaining why the quantity and^ as will be seen, also the quality 
of retrieval can vary considerably for collections having the same 
number of total term postings, in effect the same average indexing 
depth. Important also is the number of different terms posted at 
least once. The retrieval behavior of a system is different if 
many infrequent terms are deleted from the indexing of documents or 
if a few broad terms are deleted, even though the resulting index- 
ings are of like depth. 

Fewer documents are retrieved when broad, rather than narrow, 
terms are deleted, but as yet this says nothing about recall, the 
relative proportions of relevant and irrelevant documents 
retrieved. It seems plausible that a retrieval system should work 
at least to some extent. Indeed such a system would seem per- 
verse or exhausted if the retrieval of relevant documents did not 
increase as more documents were retrieved -- it would be better 
not to retrieve at all- The recall results can be explained then 



The same conclusion can be reached by considering a single 
question. A very broad term occurring in the question, for instance 
"coefficient,” can at a single match retrieve 50 documents. A 
narrow term such as "isobaric” can bring in only one document. 

Broad and narrow term deletion will affect the amount of material 
retrieved in response to this question differently. It can be 
assumed that what is true of this question is true of all questions 
-- in general, fewer documents are retrieved when broad terms are 
deleted. 
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by introducing the assumption that if a term has retrieval power 
(brings in documents) it has, when used to index a question, also 
recall power (the power to bring in relevant documents). Gen- 
erally then, comparing broad and narrow terms, fewer documents are 
retrieved when broad terms are deleted from documents, and thereby 
the largest drop in recall comes with the deletion of broad terms. 

Exceptions to the Recall Result 

There are, however, two exceptions to the general result that 
recall drops in proportion to the breadth of term deleted. The 
first exception is that recall is not in the slightest affected 
at the first cutoff level when the 4th quartile (infrequent) terms 
are deleted. This is somewhat surprising, especially when one con- 
siders that in this deletion over 1000 different index terms were 
removed from documents. Moreover, the deletion of these terms even 
at cutoff 2 does not affect recall significantly. The implication 

is that the over 1000 terms in the indexing vocabulary which have 

been posted five or fewer times are superfluous when retrieval is 
performed at low coordination levels. By way of explanation: 
broad terms cover so many documents, it seems unlikely that at a 
Coordination level as low as 1, an infrequently used term could 
bring in something new. The term **flow** for instance retrieves 
757o of the collection, while a term such as "isobaric” has at best 

the potentiality of retrieving one document out of 200 — the 

likelihood that the single document retrievable by "isobaric** does 
not also have at least a few other terms as frequent as **flow" 
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among its 34 other index terms is fairly slight. Where the 
requirement for retrieval is weak (»^g., a document need have only 
one overlap with a question to be considered retrieved), there is 
almost as good a chance that a document indexed with 35 terms will 
be matched with a given question as the same document when nine 
narrow terms are removed from its indexing. In other words, if 
retrieval is to be performed only at low coordination levels, 
there is an unnecessary redundance in indexing documents to a 
depth of 35 terms. At high coordination levels this is not so 
true. **Isobaric** is no longer redundant if the cutoff point is 
high and if its presence on a document is essential to secure the 
needed coordination for retrieval. It could be said that the 
retrieval power of broad terms is diminished and that of narrow 
terms increased as the cutoff point is raised. The higher the 
cutoff point, the more nearly the retrieval power of broad and 
narrow terms is equalized. 

The second exception to the general result that recall drops 
in proportion to the breadth of tetm deleted is at cutoff 5 where 
3rd and 4th quartile deletions give like recall. Note that al- 
ready at cutoff 3 the distinction between these, terms, i.e., nar- 
row terms (1 to 5 postings) and relatively narrow terms (5 to 13 
postings) begins to become statistically obscured (see Table 5). 
The explanation again is that at the higher coordination levels 
there is a tendency for the retrieval power of terms of different 
breadths to become equalized. Rather naturally this tendency is 
first exhibited by terms that initially differ little in breadth 
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or retrieval power. Table 5 shows that generally (there are only 
two exceptions) there is no significant difference in recall 
between deletions of neighboring quartiles: between 1st and 2nd 

quartiles; between 2nd and 3rd quartiles; between 3rd and 4th 
quartiles. 

Precision Statistics 

The precision results are shown in Tables 3,4,6 and Fig. 3 
(Appendix G) . Some caution is needed in the statistical interpre- 
tation of the values for precision. Changes in recall when index 
terms were deleted from documents were generally statistically sig- 
nificant at the .01 critical value. However, the changes in pre- 
cision resulting from the deletion of terms were fairly often not 
significant, even at the .05 critical value. ^ There is a special 
difficulty with the significance of precision values at the 
higher cutoff points. The sample sizes for precision values at 
these points tended generally to be small. Even before the 
deletion of index terms there were questions that retrieved no 
documents at all. Such questions could not be included in the 
sample. Then there were questions that retrieved some documents 
before deletion but none after, that is, precision went from some 
positive (usually small) value to 0/0. Indeterminate changes can- 

star beside a value indicates that this value represents 
a .05 significant change over the no-deletion case. A double star 
indicates that the changed value is .01 significant. An X, which 
is beside some of the precision figures, indicates that the sam- 
ple size used to determine significance was too small «6), pri- 
marily because of changes in precision which were indeterminate, 
for the statistical test to be applicable. 
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not be judged or included in judgments about differences -- they 
amount to lost cases. The small sample sizes for precision at 
the higher cutoff points is unfortunate in that the power of a 
statistical test (the probability of rejecting a hypothesis when 



The implication is that inferences about what happens to precision 
at cutoff 5, to some extent even at cutoff 4, cannot be taken as 
very reliable. 

As an illustration of the difficulty with significance at 
higher cutoff levels, see the precision value (.90) at cutoff 4, 

1st quartile deletion. This value represents an increase in pre- 
cision of nearly 1007o, yet in this case the difference in precision 
is not significant statistically. The sample size here for the 
Wilcoxen Test is 6 -- had it been any smaller no judgment could 
have been made either way as to significance. To show that there 
is a significant Increase in precision with a sample size as low 
as 6, the Wilcoxen Test requires that 6 out of 6 precision figures 
increase as a result of deletion. That is, if the outcome is to be 
only .05 probable, every question in a sample of six questions must 
show an increase in precision (actually there was an increase in 
precision in five out of the six cases). 

In the example given above, one can allow that the statistical 
test is suspect, lacking in power because of the marginal sample 
size* This seems credible, especially as a difference of from *48 
to .90 seems so large. On the other hand the precision values 
themselves might be questioned. What is the meaning of "average 



it is false) is diminished as the size of the sample decreases. 




precision** where samples are small? The precision value obtained 
at cutoff 4, 1st quartile deletion, is an average of only twelve 
values. From a quick and intuitive assessment one might say that 
the difference here over the no-deletion case seems significant 
because it looks large and the probable error is fairly small 
(+•08). But, in saying this, one is making a parametric judgment of 
sorts, a judgment about a difference in averages (means). From a 
statistical, and less intuitive, point of view it can be asked 
whether the comparison of means is a reliable indication of dif- 
ference, when the means themselves are means over a small sample. 
Especially when there is no a priori knowledge about the distri- 
bution of the means. While it can be argued that when the sample 
size is small the statistical test is not reliable, for similar 
reasons it can be argued that the average precision figures them- 
selves are suspect, N 

It can be questioned generally whether average precision is 
a good statistic to describe intuitively or otherwise the compara- 
tive retrieval effectiveness of different indexing methods. In 
cases where there are only a few questions which retrieve documents 
— such as happens at high coordination levels -- its meaningful- 
ness seems especially questionable. For instance, in Table 6 it 
can be seen that the precision differences when the term quartiles 
are deleted at cutoffs 4 and 5 are seldom significant. It can be 
argued that this reflects statistical uncertainty, rather than 
any bona fide result. Anything that is said about precision at 
cutoffs 4 and 5 must be regarded as suggestive only* 
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Precision is Never Improved by the Deletion of Narrow Terms 

Precision tends to change in proportion to the average 
breadth of term deleted from the document indexing. There is a 
high average percentage increase in precision when the 1st quar- 
tile (broadest) terms are deleted -- 867o. When the narrowest 
terms are deleted precision does not change significantly. In 
two cases, at the lower cutoffs 1 and 2, the change is not even 
observable, a consequence presumably of the already-noted redun- 
dancy of narrow terms when used in conjunction with broad terms 
at lower coordination levels. (At the higher cutoff points the 
change in precision when 4th quartile terms are deleted is observ- 
able, but not statistically significant.) 

The change affected in precision by the deletion of terms is 
generally a change for the better. Perhaps the most interesting 
result is the exception: precision is never improved by the 
deletion of 4th quartile terms, and only at cutoffs 1 and 2 is it 
improved (and then only slightly) by the deletion of 3rd quartile 
terms. This result constitutes several counterinstances to the 
"tradeoff hypothesis."^ At the same time this result gives weak 
support to the hypothesis that the narrower the terms used in 
indexing the better the precision of the results. Only weak sup- 
port because it must be admitted that the precision values obtained 
for 3rd and 4th quartile deletions are not significantly different 
from the precision values before deletion -- except in one case 

^See pp. 18-19. 
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(cutoff 4, 3rd quartile deletion). The exception may be regarded 
as suggestive of a general tendency. 

Counterinstances to the Tradeoff Hypothesis 

At cutoffs 3,4, and 5, the deletion of 3rd and 4th quartile 
terms gives results contrary to the expected tradeoff between pre- 
cision and recall. It can be asked how this could happen, how the 
deletion of terms, which decreases recall, at the same time 
decreases precision. In the random deletion case, as modeled by 
Swanson,^ precision generally improves, Swanson hypothesizes that 
the high overlap categories are denser in relevant than irrelevant 

documents than are the lower overlap categories. The net result 

2 

of deletion, at any but the highest cutoff point, is a larger 
ratio of relevant to irrelevant documents, that is, an improvement 
in precision. 

Swanson's hypothesis is borne out by the data in the present 
experiment. Nevertheless, in some cases, precision does not im- 
prove with the deletion of index terms from documents. To explain 
why precision behaves in this way, it can be observed that even 
though the conditions of the hypothesis hold, precision can get 
worse if a greater proportion of relevant than irrelevant docu- 
ments lose overlaps in the deletion process. If the deletion proc- 
ess is a random one, it is as likely for a relevant document as 
an irrelevant one to lose an overlap. But this is not true when 

^Swanson, "On Indexing Depth and Retrieval Effectiveness," 



^Ibid . 
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terms are systematically deleted from documents. Under conditions 
of nonrandom deletion the operative factor affecting the change 
in precision (for better or for worse) is the relative proportions 
of relevant and irrelevant documents that are dropped from a 
given overlap category. One cannot presume that this relative pro- 
portion will be the same for broad and narrow term deletions. 

Indeed this is just what the experiment was designed to test, viz . ^ 
whether broad terms or narrow ones function more effectively in 
the retrieving of relevant documents. As can be seen from Table 3 
and Fig. 3, when broad terms are deleted precision increases, that 
is, the ratio of relevant to irrelevant documents increases at a 
given overlap category. But the reverse happens when narrow terms 
are deleted, at least at cutoffs above 2: the ratio of relevant 

to irrelevant documents decreases and precision gets worse. 

As was mentioned, any effect narrow terms might have at the 
lower cutoff values (1 and 2) is obscured by the gross effect of 
broad terms. At the higher cutoff values, however, the effect of 
narrow and relatively narrow terms (3rd and 4th quart ile terms) 
becomes more apparent. At these values the number of documents 
retrievable by broad and narrow terms is more nearly equalized. 

When this is the case narrow terms are more effective in retriev- 
ing relevant documents than broad terms. This suggests a tenta- 
tive answer to the question originally posed, viz., which terms 
are better for retrieving relevant documents, broad or narrow 
terms? It can be said that at cutoffs above 2, narrow terms are 
at least relatively better, since deleting them does not even give 



er|c 



111 



105 



the benefit of tradeoff, both precision and recall get worse. 

This result is interesting at cutoff 3. At cutoffs 4 and 5, it 
must, of course, be regarded as unfirm. 

The Effect on Precision of Deleting Broad Terms 

The above results are given further support by the fact that 
the deletion of broad terms at cutoff 3 and above is seen to result 
in better retrieval performance, signifying thus that at these 
levels broad terms are not especially good as retrieval hooks (see 
the operating curves in Fig. 3). Intuitively it would seem that 
broad terms should be especially necessary if retrieval is to be 
performed at high coordination levels, simply to insure that there 
are some documents retrieved, i.e., that there is something in the 
intersection of document sets retrieved by the coordinated terms. 

As far as precision is concerned, this consideration is irrele- 
vant -- it is the quality of the documents retrieved that is impor- 
tant, not the fact that documents may or may not be retrieved. 
Questions that do not retrieve documents are not included in the 
evaluation of average precision. The consideration more appro- 
priately applies to recall -- however, if recall at the higher cut- 
off values had been disastrously affected by the deletion of broad 
terms from the document indexing the operating curve for the 
deletion of 1st quartile terms would never have crossed the no- 
deletion curve. Note that the operating curve for the deletion of 
broad terms crosses the no-deletion curve by virtue of three high 
precision values, those at cutoff points 3,4, and 5. The latter 




112 



106 



two values again are suspect, the one representing a seemingly 
large jump over the no-deletion case is not actually significant;^ 
the other value is obtained on the basis of a sample size so 
extremely small (2) that it is virtually meaningless. The third 
figure, however (cutoff 3, 1st quartile deletion), is the most 
stable as well as the most dramatic; it is the figure that effects 
the crossing of the operating curves; the sample size is fairly 
large; the standard error is only +.0568; the precision value 
represents one of the more highly significant differences from the 
no-deletion case, significant at less than the .01 level. Cutoff 
3 seems to be a critical point. It was observed before that the 
deletion of narrow terms below this point had little or no effect 
on precision or recall. At cutoffs of 1 or 2, retrieval is 
effected by the strength of broad terms alone. At cutoff 3, how- 
ever, the retrieval power of the low frequency terms begins to 
become apparent. It would seem from looking at the operating 
curves that the behavior of broad terms on either side of the crit- 
ical 3 cutoff is also different. At cutoff 3 or above their 
removal results in a better operating curve, but not below this 
point. Table 6 shows that the deletion of broad terms at cutoff 
1,2 or 3 results in indexing significantly better than any of the 
other indexings obtained by deleting terms, as well as no-deletion 
indexing. Moreover, the precision values are not suspect at 
these low cutoff points. The implication is that broad terms do 

^See p.lOO for a discussion of this case. 
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not make for good precision. 

Deleting broad terms is a method of improving precision. 

There is, however, another method of improving precision and this 
is to raise the cutoff point. And indeed the operating curves 
show that the deletion of broad terms at low cutoff points is not 
as effective a nove in improving precision as raising the cutoff 
point. At cutoff 1, no deletions, a 120% increase in precision 
can be bought at a 15% loss in recall by deleting the 1st quar- 
tile terms. By raising the cutoff to 2 a 240% increase in pre- 
cision can be achieved at a 16%> loss in recall. Actually, the 
^..ice paid in recall increases as the cutoff point is raised. For 
instance, at the 3rd cutoff point the deletion of these terms 
represents only an 87% increase in precision at a 56% loss in 
recall (compared to the 120%> precision and 15% recall loss at cut- 
off 1). Regardless of the precision-recall preference of an indi- 
vidual user, there is more to be gained by deleting broad terms 
at cutoff 1 than at cutoff 3. The performance curve for the 1st 
quart ile deletion is poor at low coordination levels only because 
raising the cutoff point is so much better an alternative. 

Primarily what is shown by the operating curves is the rela- 
tive usefulness of the two devices to improve precision: raising 

the cutoff point and deleting broad terms. An answer is given to 
the question: how can it be said that narrow terms make for good 

indexing when using broad terms at higher coordination levels 
might give better retrieval results2 The answer, which, of course, 
is relative to the statistical properties of the Salton-Cranfield 
collection, is that up to the critical cutoff point (3 overlaps) 
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it is indeed better to reduce the amount of material retrieved 
by raising the cutoff point. At coordination levels of 3 or above 
the more effective measure is to delete the broadest terms. The 
reasonable user will operate somewhere along the curve defined by 
the points: 1 cutoff, no deletion; 2 cutoff, no deletion; 3 cut- 

off, 1st quartile deletion; 4 cutoff, 1st quartile deletion; 5 cut- 
off, 1st quartile deletion.^ At which point on the curve the user 
decides to operate is a matter of individual preference, depend- 
ing on whether he favors precision or recall -- in effect, 

depending on how many irrelevant documents he is willing to search 

2 

through to find what he needs. 



An intuitive estimate of the distances represented between 
the points on the optimal operating curve can be obtained by de- 
scribing an imaginary situation where a user is in dialogue with 
the retrieval system. This user sets his retrieval condition at 
the 1 cutoff point and finds that he retrieves 50% of the collec- 
tion. He wants to improve precision -- it is hard to suppose 
anyone at this point not wanting to improve precision. A decision 
is made as to hew much recall can be sacrificed in the interest of 
better precision. Suppose that the user wanting more precision is 
willing to pay for it by an eq^ial (percentagewise) loss in recall. 
To improve precision then, he would be led to raising the cutoff 
point until the third cutoff level is reached, at which point the 
imperative would be to delete 1st quartile terms. He would oper- 
ate at cutoff 3, 1st quartile deletion. Any further attempt to 
improve precision would bring no improvement worth the cost, given 
of course, this user's particular tradeoff preference. 

2 

It has been suggested by Swanson ("Searching Natural Lan- 
guage Text by Computer," Science (October 21, 1960), p. 1102) that 
a decision of this sort be expressed in terms of a penalty factor, 
the "penalty" being the time spent or cost in reading irrelevant 
material. Thus the penalty factor (£) is a variable which can 
take on arbitrarily assigned values. Retrieval results can then 
be assessed in terms of a scoring algorithm, given by R - pi, where 
R and I are the respective amounts of relevant and irrelevant ma- 
terial retrieved. This scoring algorithm, which in effect converts 
precision-recall to a single measure for each user, can then be 
used to locate the best point at which to operate on the curve. 
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Comparative Effectiveness of the Indexings Resulting From the 
Quartile Deletions 

One might have expected, from the earlier discussion of nar- 
row terms, that the deletion of 4th quartile terms would give the 
operating curve closest to the leftmost corner of the graph indi- 
cating poorest retrieval. Actually it appears that at the three 
middle cutoff points the deletion of 3rd quartile terms most 
adversely affects retrieval results, with the 4th quartile deletion 
running a close second. This would suggest that, on the average, 
the deletion of terms occurring between 5 and 13 times improves 
indexing most and that therefore upper and lower limits can be 
placed on breadth. But it would be wrong to say generally that 
there is an optimal breadth which is between 5 and 13 postings 
per term. This is not demonstrable at either end of the spray of 
operating curves. Actually as far as precision alone is con- 
cerned, there is no difference between deleting 2nd and 3rd quar- 
tile terms, or 3rd and 4th, or even 2nd and 4th quartile terms at 
any cutoff point. Only the deletion of broad terms (1st quartile) 
gives results that are significant. Also for recall, the deletion 
of broad terms results in retrieval performance significantly dif- 
ferent from that of any other deletion, with the minor exceptions 
of 2nd quartile deletions at cutoffs 1 and 4. While again, for 
recall as well as for precision, 2nd and 3rd c/iartile dr^letions 
give significantly indistinguishable results. What is warranted 
by the data is that when a user wishes to maximize recall, the most 
important terms are the broader ones. If precision is desired. 
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these terms are not so important and infrequent terms seem most 
effective. 



The Effect on Expected Search Length of Deleting Broad and Narrow 
Terms 

The retrieval results evaluated in terms of Expected Search 
Length are somewhat simpler to analyze in that esl is a single 
measure of retrieval effectiveness and the results have a slightly 
different aspect. These results are shown in Tables 7,8 and 9 and 
in Fig. 4. It can be seen that, with two exceptions only, the 
deletion of index terms from documents results in larger esl val- 
ues vi.oorer performance). The general tendency can be explained 
by the rationale that the more index terms that are assigned to 
documents the more information is conveyed by the indexing and the 
greater the organizing potential of the indexings. Thus the bet- 
ter the retrieval results. This rationale, which incidently is 
an argument for broad as well as deep indexing, will be discussed 
more fully below -- it needs qualification (see pp. 127 ff.). 

Considering that the deletion of index terms generally 
results in poorer performance, one can ask whether it is possible 
to find terms, broad or narrow, which when d'elcted more seriously 
than others impair retrieval results. The inference is that these 
terms contribute most to effective retrieval. Somewhat surpris- 
ingly perhaps, it can be seen that the deviation from the no- 
deletion case which is most often significant is attributable to 
the deletion of 2nd quartile terms, those occurring between 13 and 
42 times. For three out of the 5 possible s* values, the deletion 
of 2nd quartile terms gives the largest esl values, and moreover. 
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values that are significantly different from the corresponding 
values in the no-deletion case, in the case when 4th quartile 
terms are deleted and with one exception, (s*=3), when 1st quar- 
tile terms are deleted. The surprise comes from contrasting this 
result with the precision-recall result where it is the 3rd quar- 
tile terms, not 2nd quartile terms, which when deleted give on 
the average the poorest operating curve. Third quartile deletion 
when the esl measure is used, gives significantly poorer values 
in two out of the five possible cases. 

This discrepancy, however, is not really serious. First, it 
seems one might have expected two different effectiveness meas- 
ures, even though "roughly" equivalent, to give somewhat differ- 
ent retrieval evaluations -- analogously, as two different maps, 
a geological map and a highway map present somewhat different 
aspects of the same terrain.^ Secondly, the discrepancy is 
actually insignificant. It was mentioned that as regards pre- 
cision and recall there is no difference whatsoever — at any cut 
off -- between indexings with 2nd and 3rd quartile terms deleted. 
Similarly for esl, there is no significant difference at any s* 



^One can speculate on the nature or extent of the relation- 
ship between the two measures: while it is easy to see a rough 

correspondence, the question is not easy how exactly is precision 
recall related to esl. Also of speculative interest is the rela- 
tionship between a measure and what is measured, between a map 
and what is mapped. It can be asked how well either measure 
really reflects user satisfaction as regards the utility of what 
is retrieved. The questions are philosophic, foundational 
questions. The second is especially important in that any con- 
clusions that are drawn using one of the measures are qualified 
in a rather basic way by not having an answer to the question 
"how do we know we are measuring the right thing?" 
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when terms of either these middle breadths are deleted. 

On the average the deletion of 2nd and 3rd quartile terms 
causes about a 40% increase in esl. A similar increase in 
average esl results when broad (1st quartile) terms are deleted. 
From the point of view of averages, it would seem that an argu- 
ment could be made that the main variable affecting esl is the 
number of terms posted -- whether the terms posted happen to be 
broad or narrow is relatively unimportant. Controverting evi- 
dence, however, is given by the deletion of 4th quartile terms, 
and even by the deletion of the 1st quartile terms themselves. 

The deletion of the narrow 4th quartile terms results in only an 
8% average increase in esl, as compared with the 40% increase that 
comes from the deletion of terms in other quartiles. Moreover, 
when narrow terms are deleted, the change in esl is uniformly 
slight at every value of s*^l. At s*=l the change is signifi- 
cant, but barely so. The implication is that narrow terms, those 
posted less than 5 times, are not as effective in retrieving rele- 
vant documents as terms more frequently posted. This result 
agrees fairly well with the corresponding precision-recall result. 
It was seen that the deletion of 4th quartile terms makes vir- 
tually no difference in precision or recall values at low cutoff 
levels. At high cutoff levels there was a significant difference 
in recall, but not for precision. As for broad terms: averaged 

over all s*, the increase in esl resulting from the deletion of 
1st quartile terms differs little from the average increase for 
other deletions. However, unlike other deletions, the deletion 
of broad terms gives dramatically different results for different 
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s* values. When s* is small (s* m 1,2) the deletion of broad 
terms actually improves esl. However, the improvement in esl is 
not large enough to be significant, and must therefore be consid- 



where s* is large (s* =5), the deletion of broad teims results 
percentagewise in a larger increase in esl than any other 
deletion case (57%). This extreme figure at s* = 5 counterweights 
the "anomolous" figures at s* = 1 and s* = 2, and the rather 
small deviation for middle values of s*, making thus the average 
esl for 1st quartilc deletions approximately equal to the aver- 
age esl for 2nd and 3rd quartile deletions. 

A request for many relevant documents is the esl situation 



It could be asked how esl could improve at all with the 
deletion of terms. (Analogously how could precision decrease at 
all with the deletion of terms, and the consequent drop in recall.) 
An example can be given to show that, given a small s* and the 
deletion of broad (not narrow) terms, it is not improbable that esl 
decreases. Assume that, before as well as after deletion, a re- 
quest for one document is satisfied at the topmost level of the doc- 
uments ranked by overlap. This assumption is a fairly reasonable 
one since this topmost level is densely populated with relevant 
documents to begin with, and, inasmuch as precision increases, 
becomes even more so when broad terms are deleted. Esl in the two 
cases is roughly proportional to the ratio of irrelevant to rele- 
vant documents (esl * i ) > that is, esl is roughly inversely pro- 



portional to precision. Precision increases with the deletion of 
broad terms, and esl therefore decreases. This is a possibility 
only — esl of course does not decrease if no relevant documents 
are retrieved at the topmost level after deletion. However, the 
chance of this happening is probably negligible -- from the above 
assumption. To show analytically, in the form of a theoretical 
proof, that it is either probable or improbable that esl decreases 
for s* would seem to be unmanageable. All possible alternatives 
would have to be taken into account -- requests might be satisfied 
at any overlap level and the level of satisfaction need not be the 
same before and after deletion. Moreover, it is not clear what the 
result would mean since it would have to be expressed in terms of 
expected Expected Search Length. 



ered as no more than random fluctuation.^ On the other hand 
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that roughly corresponds to the precision-recall situation where 
retrieval is performed at low cutoff points; a request for a few 
documents is the situation corresponding to a search performed at 
a high cutoff level. The data shows that this tendency for small 
requests to be satisfied at high overlap levels and for large 
requests to be satisfied at lower levels becomes exaggerated when 
broad terms are deleted from the indexing of documents. Before 
the deletion of broad terms the average level at which a request 
for 5 documents is satisfied is the two overlap level. But after 
deletion, for nearly half of the requests, the search must proceed 
down into the rather vast 0 overlap level (which is true for none 
of the potentially satisfiable 5 document requests before 
deletion). By contrast the average level at which a small request 
-- a request for 1 document -- is satisfied before the deletion 
of broad terms is the 4 overlap level, after deletion the 3 over- 
lap level. 



Indexing With Title and Weighted-10 Terms 

Depth, Breadth and Posting Statistics 

In the second stage of the experiment there were four 
deletions of index terms from documents: all index terms not 

also occurring in the title of the document were deleted and the 
same number of terms was randomly deleted; index terms that were 
not given a weight of 10 were deleted and the same number of terms 
was randomly deleted. The depth, breadth and posting statistics 
for these deletions are shown in Table 10. It can be seen from 
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the Table that approximately 20% of the index terms assigned to 
a document also occur in the document title; and approximately 
22% of the originally assigned terms are given a weight of 10^ 

The average breadth of indexing after the deletions comes to less 
than one occurrence per document. This is because so many terms, 
upon deletion, lose all their postings -- the total vocabulary 
size, however, remains unchanged.^ It can be seen that when the 
deletion of terms is systematic (the deletions of non-title and 
non-weighted-10 terms) the average breadth of deleted terms is 
6.49, whereas when the same number of postings per document are 
deleted randomly, the average breadth of the deleted terms is 
7.23. The implication is that index terms which are also weighted- 
10 terms or title terms are generally broader than terms randomly 
sampled, but not greatly so. 

Tables 11 - 15 and Figs. 5 and 6 show the precision-recall 
and esl evaluations of the indexings after each of the four 
deletions. As before, certain of the precision values at the 
higher coordination levels had to be discounted because not enough 
documents were retrieved after the term deletions to warrant com- 
parison with the no-deletion case. In addition to statistically 
comparing precision, recall and esl values for the altered index- 
ing with the corresponding values in the no-deletion case, com- 
parisons were made between paired random and nonrandom values for 
indexings of like breadth and depth. For instance, for a given s* 

^It will be remembered that BV = DN. See p. 59. 




116 



the value after title-term deletion is compared to a value result- 
ing from a ^comparable” random deletion.^ 

Weighted-10 Indexing 

Weighted-10 indexing, the indexing that results when terms 
not assigned this highest weight in the original indexing are 
deleted, gives results poorer than the original indexing, and not 
significantly better than random indexing with the same number of 
terms. At low values of s* the difference in esl from tho no- 
deletion case was .05 significant; at higher s^ values the poorer 
performance of weighted-10 indexing was even more significant (.01). 
Moreover, in no case (at no s* value) did indexing with weighted- 
10 terms differ at a .01 significance level from the corresponding 
random indexing. Precision-recall results for weighted-10 index- 
ing, when compared to the esl results, are somewhat surprising. 

They support the esl results in that when weighted-10 indexing was 
compared with a random indexing of the same depth, there was no 
case in which the difference in precision was .01 significant, and 
in only one case was the difference in recall as significant as 
this. In view of these indications so far of the inferior quality 
of weighted-10 indexing, it comes as some surprise to see that the 
precision-recall operating curves show that weighted-10 indexing 
is essentially better than the full unaltered indexing at cutoff 
point.^ 3,4 and 5. This result is not even roughly reflected by 

^In the Tables brackets are put around paired values that dif- 
fered s ignif icant ly . 
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the esl results for low s* values. In attempting to explain 
what seems like a contradiction, one might resort to the assump- 
tion that the crossing of the precision-recall curves for weighted- 
10 and no-deletion indexing was due only to random fluctuation. 

It should be pointed out that the problem of small sample size was 
particularly acute for the precision measurements on weighted-10 
indexing at cutoffs 3,4 and 5. The sample sizes at these points 
were 11, 6 and 2 respectively -- which in any case seems reason 
enough to discount the precision values at these points, viz . , the 
precision values that effect the crossing of the operating curves. 

The case for weighted-10 indexing seems then generally bad. 

One might conclude at the "semantic quality" of weighted-10 
indexing is suspect. That is, among the non-weighted-10 terms 
removed in the deletion process there were terms important in the 
retrieval of relevant documents. Alternatively one might conclude 
that because so many terms were removed in the deletion process 
(average depth was reduced from 34.5 to 7.4) , regardless of 
the quality of the weighted-10 terms left on the documents, 
retrieval effectiveness would have been seriously impaired. This 
second alternative can be rejected in light of the results for 
title -terxii indexing, also at a small average indexing depth (see 
p. 119 ff.). Weighted-10 indexing then must be judged as poor 
quality indexing. In attempting to undersrtand why, the method by 
which weights were assigned to index terms by the Cranfield index- 
ers might be reviewed. For at least 80% of the 1400 -document col- 
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lection weights were assigned to terms on a subjective basis. 

To receive a weight of 10 an index term had to be considered 

**potent** — **putent** meaning "concepts in the main general theme 
2 

of the document." Also there were other conditions to be satis- 
fied, i.e., not all potent terms were given a weight of 10. If 
the terms were very common (i.e., broad) or very vague (eg. "data") 
they were weighted less. Thus among the non-weighted-10 terms 
that were deleted were potent terms that were fairly broad and 
vague. From the point of view of the present experiment the Cran- 
field decision to exclude broad and vague terms from the highest 
importance category was not a good move. As a consequence statis- 
tics and semantics are confused, the variables are muddied. As 
was seen the deletion of broad terms can improve retrieval effect- 
iveness in the high precision regions and detract from it in the 
high recall regions. Obviously then, the exclusion of broad and 
vague terms from the weighted-10 category had the effect of over- 
valuing "potent" indexing in the high precision regions and at low 
values of s^, and devaluing it in the high recall regions and at 
high values of s*. It might be noted that even the tampered 
weighted-10 irdexing did not perform as well as the indexing from 
which the 1st quartile (broad) terms were deleted. The two index- 
ings are Incomparable in the sense that the average depth and 
breadth of the indexings are not the same. Still the Cranfield 

^Cleverdon, Mills and Keen, Factors Determining the Perfor - 
ance of Indexing Systems , p. 55. 

2 

Ibid . . p. 55. 
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policy of weighting terms might be questioned.^ Presumably a 
costly, time consuming and essentially intellectual operation, the 
weighting of terms, at least as was done at Cranfield, does not 
compare favorably with the essentially mechanical operation of not 
indexing with broad terms. This is perhaps the firmest conclusion 
that can be wrenched from the data, considering the confusion of 
mixed variables worse compounded by the discrepancy between esl 
and precision-recall values. The case for weighted-10 indexing 
neither supports nor counters the hypothesis that two indexings of 
comparable breadth differ in retrieval effectiveness because one 
consists of quality terms and the other does not. 

Title"Term Indexing 

The results for title-term indexing are more interesting. 

There were three cases \ihere the deletion of non-title terms made 
no significant (.01) change in esl values. These were at s* = 1,2, 
and 3. In one of these cases (s* = 2) the esl value actually 
improved after the terms were deleted, though admittedly the 
improvement was quite insignificant, a drop from 11.76 to 11.60. 

At s* = 4 the change in esl was significant, but only at the .05 
level, i.e., not at .01. At s* = 5 the change was significant at 
.01. The implication is that if a request is for only a few rele- 

^Salton finds that weighted word stems extracted from docu- 
ment abstracts are clearly more effective than non-weighted stems 
(see Salton, ''Computer Evaluation of Indexing and Text Processing," 
p. 24). However, he used automatic weighting methods rather than 
the weights assigned by the Cranfield indexers, those representing 
intuitive assessments of semantic relevance. 
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vant documents, it is unnecessary to index as deeply as 34 terms 
per document -- seven terms would do as well, if these terms are 
title terms; however, if a request is for many or all relevant 
documents, indexing to a depth of seven terms per document is not 
sufficient, no matter what the quality of the terms. More terms -- 
more information -- is needed to partition the collection for 
effective retrieval. 

Comparing title terms with a set of terras chosen randomly from 
the original indexing to a like depth (breadth), it can be seen 
(Table 15) that at every s* value there is a significant difference 
in performance. In every case, title terms are significantly (.01) 
better for retrieving documents relevant to the search questions. 
(By contrast, in no case does weighted-10 indexing give .01 signif- 
icantly different esl values than random indexing of the same 
depth. ) 

Evaluation in terms of precision and recall shows that at the 
higher cutoff levels (roughly corresponding to low values of s*) 
the operating curve for title-term indexing is better than the 
curve for the no-deletion case. Also corroborative of the esl 
results for title-term indexing is the precision-recall perform- 
ance of title terms compared with a random sample of terms chosen 
from the full indexing to a comparable depth. At every cutoff 
value the difference in recall between the two indexings is .01 
significant. The difference in precision, however, is as signif- 
icant as this at only one cutoff point (cutoff 1) but at another 
(cutoff 3) it is .05 significant. (By contrast weighted-10 index- 
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ing compared with the corresponding random indexing showed a ,01 
significant difference in precision in no case, and in recall in 
only one case.) 

If the average breadth (depth) of indexing were the main 
variable affecting retrieval performance, one would expect similar 
effects whether terms were deleted randomly or on purpo^.e. The 
hypothesis might be stated that particular words in themselves are 
not important, more important is the actual number of words chosen 
to index the documents. The results for title-term indexing in 
tiie high precision, low s* regions constitute a counterinstance to 
this hypothesis. It would seem that title terms are quality terms 
in the sense that they are especially effective in retrieving rele- 
vant documents. From this one might infer that there is a cor- 
relation between the title of a document relevant to a given 
question and the fact of its relevance. There is a difficulty 
here however. It is not clear that the correlation is a real one. 
It is real only if there is no unnatural bias between a question 
and the title of a relevant document. An unnatural bias might come 
about if, for instance, in the original assessments of relevance, 
some relevant documents, perhaps those with unobvious titles, were 
overlooked. Very likely this is what happened. The fact that 
students did the searching for relevant documents, and no doubt 
their attention was flagged by noticeable titles, makes it prob- 
able. Cleverdon was concerned about the problem of unnatural bias 
and in an attempt to determine whether one existed he examined 
some of the search questions together with their relevant docu- 
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ments. He found that nearly 1/3 of the questions had a strong 

1 

question-title match. But the figure of 1/3 can neither prove 
nor disprove the existence of an unnatural bias. Cleverdon*s 
words are more persuasive: ”one would have expected a certain 

strength of title match in this subject, where titles are usually 

2 

fairly long and a good indication of the subject of the document," 
and "it is wrong to conclude . . . that whenever there is a sub- 
stantial match between question and title, then the relationship 
is necessarily unnatural." This seems fair and Cleverdon admits 
that nothing conclusive can be said on the subject of a title- 
question bias until questions are obtained from a real life situa- 
tion and tested in an existing collection. Indeed one might spec- 
ulate that many "real life" questions are really searches for 
partially remembered titles, in which case the correlation between 
question and relevant-document title would be strong and natural. 

As regards the results of the present experiment, however, it needs 

and perhaps suffices to be said that they are limited by the 

4 

assumption of no unnatural title-question correlation. 



^Cleverdon, Mills and Keen, Factors Determining the Perfor - 
mance of Indexing Systems , Vol. 1, 38. 

2 , 

Loc . cit . 

3 

Cleverdon, Mills and Keen, Factors Determining the Perfor - 
mance of Indexing Systems , Vol* 1, 36. 

^Salton's results for title-term indexing using this data 
might be mentioned, Salton compared the automatic indexing of 
titles with the indexing of abstracts, also done automatically. 

He finds the abstract process to be superior to the title-only 
option. This result, while important from the point of view of 
the feasibility of machine indexing, does not say much one way or 
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the other about the special usefulness of words which occur in 
titles. This is because the titles were indexed to an average 
depth of 11 descriptors, while the abstracts were indexed to the 
much greater (average) depth of 91; so what Salton shows is that 
with 91 descriptors mechanically derived from an abstract it is 
possible to achieve better results than with 11 descriptors 
mechanically derived from a title. It seems quite possible that 
the superior performance of abstract indexing can be accounted for 
by the greater "depth" of indexing. At least there is reasonable 
doubt that Salton* s experiment demonstrates anything conclusive 
about the innate value of the particular words used in indexing. 




130 



CHAPTER IV 



CONCLUSION 



The Case for Specific Indexing 



The main purpose of the experiment was to answer the question 
which term functions most effectively in the retrieval of rele- 
vant documents, broad or narrow. The answer, as has been seen, is 
not straightforward, but depends upon what the user of the 
retrieval system wants, his wants being expressed in terms of a 
stated precision-recall preference, or, more precisely, by the 
exact number of relevant documents he wishes to retrieve. This 
result is not unexpected. At the end of the last century Cutter 
advised that the principle of specific entry be viewed in light of 
another principle, viz . . the usage principle. Over the years 
there has perhaps been too much concern about the **language” of 
the users, at least the present study shows that an important oper- 
ational factor is the user's quantitative needs. Some insight 
in:o the nature of the relationship between indexing specificity 
and user needs can be given by summarizing the experimental 
results as they bear on indexing with narrow (specific) terms. 

The deletion of narrow terms (1-5 postings per term) affects 
recall less than the deletion of broad terms. Precision is never 
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improved by deleting narrow terms. When retrieval is performed at 
high coordination levels (in the present study at coordination 
levels greater than 2), the deletion of narrow terms adversely 
affects retrieval output with respect to precision, as well as 
recall:. Even-the deletion of relatively narrow terms (5-13 post- 
ings per term) decreases precision at these levels. By contrast, 
the deletion of broad terms improves precision, at any coordina- 
tion level, and the operating curve for the deletion of broad terms 
is better than the curve for the no-deletion case at coordination 
levels greater than 2. The quality of retrieval in either preci- 
sion or recall is not affected when narrow terms are deleted at low 
coordination levels^ at cutoffs of 1 and 2. Narrow terms, while 
important where high precision is desired, are redundant and 
unnecessary when the demand is for high recall. Retrieval eval- 
uation measured in terms of esl generally supports these conclu- 
sions. The deletion of narrow terms does not affect esl values 
significantly except at the lowest request level where the user 
wants only one relevant document, in effect where the user wants 
high precision. 

An argument can be made for the "importance" of narrow terms 
in indexing for quality retrieval. What happens as the cutoff 
point is raised is that the retrieval power of broad and narrow 
terms becomes equalized, that is, the difference between the num- 
ber of documents a term brings in becomes obscured, while the dif- 
ference in the quality of the retrieved documents with respect to 
relevances becomes more apparent. The fact that at higher coordi- 
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nation levels the deletion of narrow terms makes for worse preci- 
sion as well as worse recall suggests that the documents retrieved 
by narrow terms are more likely to be relevant than those retrieved 
by broad terms. This difference between the retrieval effective- 
ness of broad and narrow terms is not fully explainable on the 
basis of the relative sizes of the document sets retrieved, since, 
in its demonstration, the tradeoff hypothesis was violated. In 
other words, narrow terms are essentially good index terms, and 
not just apparently so by virtue of statistical accident. 

The argument for narrow terms must, however, be qualified. 
There are cases where index terms can be too narrow. If a user is 
interested in high recall or needs more than one relevant docume*:it, 
narrow terms, though "quality" terms, are ineffectual, the work of 
retrieval depending on broad terms. It was shown that on the aver- 
age -- i.e,, averaged over different request sizes and different 
cutoff values -* index terms that are posted between 5 and 32 times 
are "optimal" judging by the effect of their absence on retrieval 
performance. One is reminded of Pettee*s "guess" of "less than a 
dozen" postings per term as the lower limit on specificity.^ It 
is misleading, however, to speak of index terms of "optimal 
breadth," since the best breadth of term is a variable -- it varies 
with the user's decision as to the number of relevant doctiments he 
needs or the number of irrelevant documents he can tolerate. Lil- 
ley's answer to the question "How specific is 'specific'?" is 

^See p. 22. 
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sound. **Well, it all depends!”^ But the answer was left hanging. 

Lilley does allow it depends on user’s needs, but intangibly so: 

^Specificity may be so intangible as to be nothing more in an 

effective sense than a chance relationship between the user’s need 

of the moment and the format of a particular book an individual 

2 

library happens to own/* Specificity need not be chancy or 
intangible whan user’s needs are interpreted in terms of a 
precision-recall preference or the number of documents desired. 



O 
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The Case for Indexing in Depth 

The second purpose of the experiment was to examine the ques- 
tion how important a factor in retrieval effectiveness is average 
breadth or depth of indexing. It was speculated that perhaps the 
quality of particular index terms was more significant in the 
retrieval of relevant documents than statistical properties of the 
indexing such as the number of index terms assigned to a document 
or the number of different terms used in indexing the collection. 
The experimental results that are most interestingly relevant to 
this question are those for title- term indexings It was seen that 
when only one or two documents on a topic were requested, indexing 
with seven terms taken from the document title, presumably seven 
quality terms, led to good retrieval performance: in some cases 

performance results were slightly better, but in no case were they 
significantly different than indexing with the full 34.5 terms, 

^Lilley, "How Specific is ’Specific’? " p. 8. 

2 

Ibid . , p. 6. 
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including title terms, originally assigned to each document. On 
the other hand, when a request for four or five documents was put 
to the system, citle-term indexing did not perform so effectively, 
again underlining the importance of the variable "user's needs, 
quantitatively expressed." 

Cleverdon in the Cranfield II study experimented with title, 
abstract and full document indexing. In explaining the differ- 
ences in the results for these three types of indexing he cites 
indexing depth (exhaust ivity) as the "main operational variable." 

There is the possibility that the selection 
of terms by the indexer was more descriptive 
of the document content than those terms used 
for the titles and the abstracts, but the main 
variable in these five results concerns the level 
of indexing exhaustivity . It would seem that 
while the titles were at too low a level of 
exhaustivity (7 terms per document), the gradual 
increase in the level, up to an average of 33 
terms, brought about an improvement in perform- 
ance. However, the higher level of exhaustiv- 
ity represented by the abstracts (probably 
about 60 terms per document) was too high, 
resulting in the retrieval of large nYmbers 
of additional non-relevant documents. 

Cleverdon' s opinion might be questioned. The small differ- 
ence observed between title-and full-document indexing (59.767o 
and 65.007o normalized recall ratios respectively) may not be 
statistically significant. Even if significant it seems a little 
surprising that so small a difference in retrieval performance can 
be explained by so large a difference in depth — 7 terms per 
title ys. 33 terms per document. Moreover, if operational depth 



^Cleverdon and Keen, Factors Determining the Performance of 
Indexing Systems , Vol. 2, 259. 
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is an important factor in retrieval performance one would expect 
similar effects from indexings of the same depth. But the present 
experiment offers a counter-instance: title-term indexing proved 

significantly better than the "random** indexing at a comparable 
depth. Cleverdon, to support his contention that depth is the 
main variable, observes that retrieval performance improves with 
the gradual increase in depth from seven terms per document up to 
33 terms per document. The present experiment shows that title- 
term indexing at an average depth of seven terms per document 
gives a better precision-recall operativig curve than some indexings 
of 25 terms per document (when the quartile deletions were made). 
The expression **the main variable** seems somewhat incautiously 
used. Clearly retrieval performance is a function of the quality 
of the index terms assigned to a document as well as their quan- 
tity. On the other hand, it cannot be denied that the weakness of 
title-term indexing is its lack of depth. As was seen, this is 
especially apparent when the number of documents requested of the 
system is large, when high recall is desired. The first person to 
make a principle of title-term indexing realized its weakness. 
Crestadoro in his Manchester Catalog , a concordance of titles, 
warns his users that in using the catalog some **relevant** items 
will be missed: **under any given subject the whole of the books 

. . . are not brought together, but only those in which the name 

of the subject occurs in the title# **^ Of course, Metcalfe regards 
\ 

Quoted in Metcalfe, Information Indexing , p. 48. 
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the weakness as fatal and insists that title-term indexing is a 

delusion, which, while he can understand why it may have arisen, he 

1 

fails to see why it persists. It perhaps has as much reason for 
persisting as "specific” indexing. Either indexing seems ade- 
quate for retrieval situations in which completeness of recall is 
not necessary, or, for economic reasons, not possible. Situations 
where not all relevant documents on a subject are needed allow for 
compromise -- the number of documents that can be retrieved on a 
title basis may be all that is required. 

The fact that the "weakness" of title-term indexing is its 
lack of depth raises the question whether there is an optimal 
average depth (or breadth) of indexing. The data seems to warrant 
the opinion that, for the most part, the deletion of terms from 
documents affects retrieval performance adversely. This seems 
especially true when performance is measured in terms of esi. It 
might be said to be qualifiedly true when the precision-recall 
measure is used, in the sense that the better results obtained by 
the deletion of terms are not always credible since they occur at 
high cutoff levels where the sample size is usually small. One 
might hypothesize that the indexing of documents should be as 
deep (broad) as possible, and examine the hypothesis for its plau- 
sibility. 

To support the hypothesis an example can be given, illustrat- 
ing that even the addition of a synonomous term to the indexing of 
a document could improve retrieval performance. Let s* = 3 and 

^ Ibid , , p. 49. 
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assuine the following two-level ordering, the first (case 1) having 
the document indexed without the synonomous term, case 2 with the 
synonomous term: 



Assume also that the question, either by the user or by the system 
via a thesaurus, is indexed with the two synonomous terms, Tn this 
case then the addition of the synonomous term to the t'oeumont index- 
ing has the effect of giving a relevant document an additional over- 
lap with the question (case 2). Esl in chc second case is 2,25; 
in the first case, before the addition of the synonomous term, it 
is 7.5. However, theoretically it can be argued that tliore is an 
upper limit on this redundancy effect, i.e., there Is an upper limit 
to the number of terms which optimally should be assigned to a doc- 
ument, One can imagine an extreme case where the indexing is as 
deep as vocabulary size permits, where each document is indexed by 
every term on the indexing vocabulary. In this extreme and limit- 
ing case the document collection would be as unstructured and dis- 
ordered as had there been no index terms assigned at all. In 
either case the collection would be searched for relevant docu- 
ments, as though at random, the only difference being that in the 
one case all documents are collected at the highest overlap cate- 
gory and in the other at the 0 overlap category. Neither way is 
the indexing discriminating or informative. It is possible then, 
to index so deeply as to get no information at all about which doc- 
uments are relevant to which questions. Hypothetically at least. 



2 overlaps 
1 overlap 



Case 1 OOtOt 
00000 # 



Case 2 # 0#0#0 

00000 ■' 
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esl need not increase with the deletion of index terms. On the 
other hand, though the point of effective redundancy in indexing 
can be over-reached,^ it would seem that the content of the docu- 
ments being indexed, as well as language itself, would set a nat- 
ural limit on possible redundancy far below this limit. 

The operational imperative suggested by the foregoing is to 
index as deeply as semantic meaning will allow. But this is spec- 
ulation only. Nothing in the experiment really justifies a state- 
ment about an upper depth limit, since the experiment was 
restricted by the number of terms originally assigned to the doc- 
uments (on the average 35). It might be suggested that the exper- 
imental results indicate a practical limit to indexing depth, in 



^The question about the amount of redundancy needed in index- 
ing can be regarded as a question about the distribution of index 
terms. In the literature it has been suggested that the most in- 
formative distribution of index terms is the one that makes for the 
most uncertainty about the number of postings any given terms will 
have (see Pranas Zunde and Vladimir Slamecka, "Distribution of 
Indexing Terms for Maximum Efficiency of Information Transmission," 
American Documentation ^ XVIII, (April 1967), 104). It is not clear. 
In fact, it is unlikely that semantics can be so easily dismissed. 
Perhaps che truest thing that can be said is only tautological: 
what makes indexing informative is its discriminating power, its 
ability to structure the collection in such a way that Lt is easier 
to retrieve relevant documents than irrelevant ones. Conceivably 
the collection should be structured differently depending on 
whether requests for relevant documents are small or large. One 
might say, with regard to the example above, that of h single doc- 
ument request, that the structuring of the collection is overin- 
formative, the high overlap levels being too rich in relevant doc- 
uments so that esl could be improved by term deletion. It could 
be regarded as under informative when the number of documents re- 
quested is much larger than the number retrieved at any overlap 
level above 0. In any case it does not seem that optimal or norma- 
tive index term distribution can be discovered by statistical 
information theory (Zunde and Slamecka) or, for that matter, by a 
descriptive or mathematical analysis of how index terms are 
actually distributed (Wall). In some manner questions about the 
depth, breadth and distribution of index terms must be related to 
measures of retrieval effectiveness. 
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the sense of diminishing returns; if the deletion of terms does 
not give performance values which are significantly worse, the 
cost of assigning these terms at the time of the original index- 
ing is not really justified. The esl results show that for any 
size request at least one quartile of the index terms assigned to 
documents do not contribute significantly to effective retrieval. 
Unfortunately it is not always the same 25% of the postings which 
are superfluous « 



Practical Implications 

Library practice over the years has attempted to reach a com- 
promise between good precision and good recall. It will be remem- 
bered that it was the lack of precision in retrieval output that 
first title-term indexing and then the alphabetical catalog with 
its specific entry principle was intended to remedy. Consciously 
and perhaps unconsciously there has been an unwillingness to com- 
promise. Unconsciously in that there seems to be an attempt to 
obey two conflicting imperatives: to bring all books on a subject 

together (recall) and to index as specifically as possible (pre- 
cision). One can read this conflict in the history of the contro- 
versy over alphabetical and classified catalogs. Unwillingness to 

compromise is also represented in the conscious opposition to the 

2 

principle of double entry, of having both broad and narrow terms 

^See for instance Metcalfe, Information Indexing , p. 48. 

2 

See the discussion beginning on p. 19. 
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to serve both sorts of users, those wishing completeness, and those 
needing only a few relevant documents. The main argument against 
using double entries has been that the catalog would be burgeoned 
with an unwieldy reference apparatus disproportionate to the num- 
ber of actual items processed. A more persuasive argument 
against , perhaps, is the cost and time required by comprehensive 
indexing, when budgets and personnel are limited. 

A corollary result of the precision-recall evaluation of broad 
and narrow terms is that a method is given for determining the rel- 
ative effectiveness of the two precision devices: raising the cut- 

off point and deleting broad terms. For the collection studied, 
raising the cutoff point (making the search more exhaustive) is 
more effective until cutoff 3 is reached, at which point the best 
means to precision is to delete the broadest terms assigned to the 
documents (making the search more specific). This is an interest- 
ing result and if generalizable could have practical implications 
for libraries of the future. Ideally libraries should be able to 
accommodate request of any recall-precision requirement; in terms 
of esl, requests of any size. Documents should be indexed with 
broad terms to satisfy recall preferences, with narrow terms to 
satisfy precision preferences. It has been suggested^ that index- 
ing with narrow terms (as specifically as possible) would be suf- 
ficient, since one can always "explode** on these terms -- presum- 
ably via a thesaurus or classified schedule be led to the 

^See F.W. Lancaster, **MEDLARS; Report on the Evaluation of 
Its Operating Efficiency,** American Documentation , XX, (April 1969), 
131. 
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appropriate broad terms quite automatically. This perhaps shows 
an overassuming confidence in the simplicity of language structure 
as represented in thesauri. In any case it requires less assump- 
tion about the exploding possibilities in language to say that 
documents should be indexed with broad and narrow terms, and as 
deeply as possible up to the limit imposed by semantic common 
sense. It need not be said of an index language that it is too 
exhaustive, that there are too many ^'concepts** indexed; or that 
it is too specific -- in an ideal system at least, a narrow term 
does not preclude a broad term also being assigned. If retrieval 
values should be poor, it can always be argued that the fault lies 
not in the indexing but in the search strategy used. An ideal 
system allows the possibility of custom retrieval, of indexing 
that can be adjusted to every user's need, simply by varying search 
strategy. From the point of view of precision-recall, a search 
canbe varied by deleting broad or narrow terms, or by changing the 
cutoff condition required by retrieval. The operating curves of a 
system prescribe a search strategy for each individual user, once 
he has decided on the particular balance he wishes maintained be- 
tween precision and retr,ll. From the point of view of esl, this 
balance is, in effect, achieved by specifying the number of rele- 
vant documents requested of the system -- this is part of "search 
strategy." In either case, whether the effectiveness of a re- 
trieval system is measured in terms of precision-recall or esl, it 
would seem that, ideally at least, the best indexing is an index- 
ing which is potentially all possible Indexings and which, as part 
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of a clearly explicated search strategy, is realized as a partic- 
ular indexing, broad or narrow, as the case requires. Thus, it 
is not a question of deciding in the abstract which method of 
indexing is "best," but of giving the user at the time of search 
the option of tailoring the indexing to his needs. 

Indexing as deeply and broadly as document content permits is 
perhaps rather much of a theoretical luxury. From the point of 
view of traditional indexing practices it would be out of the 
question. Indexing depth, where specified at all, has been deter* 
mined not by theoretical considerations but by policies dictated 
by cost and the practicality of catalog size# It is not likely 
that economic problems will vanish. Conceivably, automatic index- 
ing procedures will be developed that will make the ideal of com- 
prehensive indexing realizable. There are difficult problems in 
this area, however: problems in semantics, similar to those be- 

setting mechanical translation, and problems in the technology of 
developing optical scanning devices. The old question of burgeon- 
ing a catalog with too many entries can be reformulated as a 
question of burgeoning a computer memory. Problems of file struc- 
ture and search codes are exceedingly complex and mind-boggling. 

It would be surprising, for instance, if an optimum file structure 
(ordering of entries) is less elusive a goal than a single optimum 
breadth or depth of indexing. Ideally, entries in a file should be 
ordered differently for each request to make possible a systematic 
search for relevant documents. The number of entries in a file 
such as the catalog of a large university library can easily run 
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into the millions* As yet it is economically unthinkable that 
all entries in this file could be weakly ordered according to over- 
lap each time a new question is to be searched. It rather seems 
as though it will be awhile before the time is ripe for personal- 
ized retrieval, in which case the need for compromise still exists. 
The present study, while it suggests an operational rule for speci- 
fication indexing for ideal libraries of the future, is not in 
itself sufficient to deal with the present and real problem of com- 
promise which faces libraries today. What is needed in addition 
to such a study is information about user preference, for example 
a preference for precision over recall (or vice-versa); or, in the 
case of esl, a preference expressed in terms of the number of doc- 
uments needed to satisfy a request. The difficulty here is in 
trying to approximate what perhaps does not exist, viz . , a typical 
or average request for information. 

Summary 

By way of conclusion, certain limitations of the present 
study can be mentioned. They have been referred to in the text, 
and can be expressed here in the form of unknowns: 1) not knowing 

the effect on retrieval performance of the subject matter of the 
test collection, p. 73; 2) not knowing whether the sample size 

is large enough or sufficiently random to yield generalizable 
results, p. 78; 3) not knowing, with statistical certainty, the 

behavior of precision at high coordination levels, p. 99; 4) not 

knowing whether the coordination between questions and the titles 
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of documents relevant to these questions reflects a bias one might 
expect to find in actual non- experimental retrieval situations, 
p. 121 ff.; 5) not knowing what a **typical" question or search 
request is, p, 137. 

It might be said that to a certain extent the experimental 
results are circular in nature. The specificity or breadth of a 
term is defined as the number of documents indexed by the term; 
and retrieval effectiveness is evaluated with respect to the num- 
ber of relevant documents a user wants or the number of irrelevant 
ones he will tolerate. There are, however, certain findings of 
the experiment that are not obvious: 1) deleting narrow terms 

never improves precision -- a counterinstance to the tradeoff 
hypothesis, in that deleting narrow terms can make both precision 
and recall worse; 2) the dichotomy between improving precision by 
indexing with narrow terms or by using broad ter,.y^s and raising the 
cutoff point is a false one — narrow terms are more effective as 
retrieval hooks at high cutoff levels than they are at low levels 
where, in fact, they are redundant; 3) the deletion of broad terms 
is in some instances a better means for improving precision than 
raising the cutoff point. 

The significance of the present study lies in its being an 
attempt to use objective criteria for evaluating the "goodness" of 
indexing in terms of its specificity. Specificity, as a factor in 
indexing, has been considered important since the middle of the 
last century, yet till now there has been only speculation, com- 
bined though with firm conviction, about its certain usefulness 
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and uncertain meaning. The designing of the experiment to study 
the effect of varying levels of specificity on retrieval perform- 
ance has forced an explicit definition of "specificity" and has 
required an examination of specificity in its interrelatedness 
with other factors affecting retrieval performance. I would like 
to regard the study more as a method of historical and experimental 
analysis than as an attempt to provide operational answers -- it 
would have been surprising if there were easy answers. Qualifiedly 
then, the findings of the experiment may be summarized the follow- 
ing way: 

1. The amount of material retrieved in a system is not a 

simple function of the total number of terms posted to documents in 
the collection. This is perhaps obvious, but it is useful in ex- 
plaining why the quantity and quality of retrieval can vary con- 
siderably for collections having the same number of total term 
postings, in effect the same average indexing depth. Important 
also is the number of different terms posted. This is evidenced by 
the results for indexings of like depth that show different 
retrieval behavior depending on how terms were deleted from docu- 
ments: whether many infrequent terms were deleted or a few broad 

terms were deleted. 

2. At high cutoff values the retrieval power of broad and 
narrow terms tends to become equalized. The stricter retrieval con- 
dition at high cutoffs -- eg., a document must have five overlaps 

in common with a document to be considered "retrieved" -- has the 
effect of constraining the amount of material any single term. 




146 



140 



especially broad terms, can retrieve. Where the cooperative 
strength of several terms is required for retrieval, the exist- 
ence of narrow terms on documents becomes important* On the other 
hand, at low cutoff values, narrow terms appear to be redundant in 
that the broad terms cover so many documents — eg., flow which 
indexes 75% of the collection -- it is unlikely that a narrow or 
infrequently used term could bring in something new. 

3. At higher coordination levels, i.e., when precision is 
desired, narrow indexing is preferable to broad indexing. At these 
levels the removal of broad terms from the document indexing im- 
proves the quality of the retrieval output as seen from the operat- 
ing curves for the various indexings, while the removal of narrow 
terms affects both recall and precision adversely. At lower coor- 
dination levels, i.e., where recall is maximized, the amount of 
retrieved material is so overwhelming that narrow terms are redun- 
dant and insignificant in effect. The removal of broad terms, 
while stemming the flood, is not as effective a measure for reduc- 
ing the amount of irrelevant material retrieved as raising the 
cutoff point. What is most obvious in the data is that optimal 
breadth (and optimal depth as well) seem to be variables depending 
on cutoff point, ultimately on users* preference for precision or 
recall. 

4* The deletion of index terms from documents generally 
impairs retrieval performance when measured by esl. On the whole 
relatively broad terms, terms occurring in the 2nd quartile 
(those posted 13 to 32 times), have the most power to retrieve 
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relevant documents, as measured by the increase in esl when these 
terms are deleted from the indexing, fourth quartile terms, those 
used most infrequently, are not especially effective in retrieving 
relevant documents and for requests for more than one document 
they can be dispensed with without causing significantly lower esl 
values. The broadest terms are important retrieval hooks when the 
number of documents requested by the user is fairly large; they are 
not significantly important when the number of documents requested 
is small. As was seen in the case of precision and recall, what 
is "optimally** the best breadth of term is a variable -- it varies 
with cutoff point and the user's decision as to how much irrelevancy 
he can tolerate. When retrieval effectiveness is measured in terms 
of esl, optimal breadth varies according to the number of relevant 
documents the user wishes to retrieve from the system. 

5. The operating curves for precision-recall show the rela- 
tive usefulness of the two devices for improving precision: rais- 

ing the cutoff point and deleting broad terms. This is presumably 
something that will differ from collection to collection. For the 
Cranfield II data, raising the cutoff is a better method for reduc- 
ing the amount of irrelevant material retrieved until a cutoff of 

3 is reached. After this "critical value,** the more efficient 
method for improving precision is to delete broad terms. 

6. The results for highly weighted indexing are disappoint- 
ing. Indexing with the subset of weighted-10 index terms, those 
judged most highly descriptive of the text, resulted in signifi- 
cantly poorer retrieval performance than indexing with the full 
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set of originally assigned terms. This was true for performance 
measured in terms of precision-recall at low cutoff points and 
when the esl measure was used, for all s* values. Moreover, index- 
ing with weighted-10 terms differed insignificantly, in retrieval 
results produced, from indexing with a subset of the same number 
of terms chosen at random from the original indexing. In fact, it 
appeared that better retrieval performance could be achieved by 
mechanically deleting the most frequent terms from the original set 
of terms than by conscientiously selecting out seemingly "unim- 
portant" terms. However, a conclusion that a policy of weighting 
terms is not efficient should be suspended; it is possible that the 
fault is peculiar to this experiment and lies in the Cranfield II 
rules for weighting terms, especially the rule that says broad and 
vague terms should be excluded from the highest weight category. 

7. The results for title-term indexing are less inconclusive. 
At high cutoff or low s* values, indexing with seven terms taken 
from the title of a cocument resulted in retrieval performance not 
significantly different from that obtainable from the original doc- 
ument indexing consisting of five times that many terms. Moreover, 
there was a significant difference in performance between indexing 
with the same number of terms chosen randomly from the originally 
assigned terms. On the other hand, the limits of title-term in- 
dexing are realized in those cases where user requests are for 
many documents, at the high recall region of the precision-recall 
operating curve, at high values of s*. The title- term results, 
thus, qualifiedly support the hypothesis that two indexings of 




143 



comparable breadth or depth differ in retrieval effectiveness 
because one consists of quality terms and the other does not. 
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APPENDIX A 



COMPLETE VOCABULARY LIST 
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01 


abel 


01 


00 


aberration 


00 


00 


ablated 


00 


01 


ablatine 


01 


O'l. 


ablation 


00 


00 


able 


00 


01 


about 


00 


01 


above 


00 


01 


abrupt 


00 


00 


absence 


01 


01 


absolute 


01 


00 


absorbed 


01 


01 


absorption 


01 


01 


accelerated 


00 


01 


accelerating 


03 


05 


acceleration 


02 


00 


accelerators 


01 


00 


accelerometer 


00 


00 


acceptance 


00 


00 


accidental 


00 


01 


accommodation 


00 


00 


a c cumula t i on 


01 


00 


accuracy 


00 



accurate 

acetate 

ackeret 

acoustic 

acoustically 

across 

a c tine 

action 

activation 

active 

activity 

actual 

addition 

additional 

adiabatic 

adjacent 

adjustable 

adjustinfy 

adsorption 

advance 

advancing 

adverse 

aeolotropic 
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37 


aerodynamic 


00 


airliner 


00 


aerodynamically 


00 


airload 


00 


aerodynamics 


01 


airspeed 


00 


aeroelastic 


00 


airstream 


01 


aeroelasticity 


00 


airy 


16 


aerofoil 


00 


alclad 


00 


aeronautical 


00 


alcoa 


01 


aeroplane 


00 


alden 


00 


aerothermochemistry 


00 


alfven 


01 


aerothermodynamic 


00 


alp:ebraic 


00 


aerothermoelastic 


00 


algorithm 


00 


affected 


00 


aligned 


00 


an 


00 


alignment 


00 


after 


00 


all 


06 


a-^terbody 


01 


allmovable 


00 


afterburner 


00 


allmoving 


00 


afterburning 


00 


allowable 


00 


aftercooler 


01 


alloy 


00 


afterflow 


00 


almen 


02 


ahead 


00 


alone 


01 


aileron 


00 


along 


32 


air 


00 


alteration 


Oil- 


aircraft 


01 


alternating 


00 


aircraftresearch- 

association 


00 


alternative 


00 


airflow 


16 


altitude 


00 


airframe 


01 


aluminum 


00 


air jet 


03 


ambient 
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00 ames 
00 ammonixim 

02 amount 

00 amplification 

01 amplitude 
00 analoqrous 

00 analopTue 

01 analoiry 

00 analyser 
19 analysis 

03 analytic 
06 analytical 

01 anemometer 
^■'•'6 anc^le 

00 anyled 
00 ans^ular 
00 angularly 
00 anhedral 
00 anisotropy 

00 annular 

01 annulus 
00 antenna 

00 anticlastic 

01 anti symmetric 

00 anti symmetrical 

01 apex 

02 apOf^ee 

01 apparatus 



01 apparent 

01 appearance 

00 appli cabili ty 

00 application 

02 applied 

01 approach 

01 approaching 
00 appropriate 
09 approximate 
00 approximating 
11 approximation 

00 ar 

08 arbitrary 
06 arc 

01 ardc 

03 area 
01 argon 

00 around 

01 a r ra ngeme n t 

00 arrest 

01 arrow- 
00 art 

02 artificial 
00 artificially 
00 asbestos 

00 ascending 
00 ascent 
00 asme . 
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20 

00 

Cl 

00 

00 

00 

00 

00 

01 

00 

00 

06 

00 

01 

21 

06 

01 

02 

04 

02 

29 

00 

02 

I 00 

00 

02 

CO 

ERlCt 

f 



aspect 


00 


assessment 


00 


associated 


00 


assumed 


00 


assuming 


01 


assumption 


01 


astrolite 


00 


astrophysics 


02 


asymmetric 


00 


asymmetrical 


00 


asymptote 


01 


asymptotic 


12 


asymptotically 


01 


atlas 


06 


atmosphere 


06 


atmospheric 


00 


atom 


01 


atomic 


00 


attached 


00 


attachment 


01 


attack 


00 


attenuating 


00 


attenuation 


03 


attitude 


00 


audio 


02 


augmentation 


01 


aural 


01 
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auto 

autocorrelation 

autocorrelogram 

autoignition 

automatic 

autopilot 

auxiliary 

average 

averaging 

avoidance 

avro 

axial 

axially 

axis 

axi symmetric 

axi symmetrical 

back 

backing 

backward 

baffles 

bakanov 

bakelite 

balance 

ball 

ballistic 

ballotini 

balsa 
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01 


bands 


00 


00 


bandtd.dth 


08 


01 


baagbang 


01 


00 


bank 


00 


00 


banked 


00 


00 


bar 


00 


00 


bare 


00 


00 


barrel 


00 


06 


base 


00 


00 


based 


02 


01 


basic 


00 


00 


batdorf 


00 


00 


bays 


00 


01 


bead 


00 


02 


beam 


00 


00 


beane 


00 


00 


bearing 


00 


00 


beat 


00 


00 


bed 


00 


00 


bedford 


00 


00 


b92:inning 


01 


0^!. 


behavior 


01 


06 


behind 


01 


00 


bei 


00 


00 


bell 


00 


00 


beUeville 


00 


00 


belotserkovski 


13 


01 


belt 





bend 

bending 

beneath 

ber 

bernoulli 

berthelot 

bessel 

bestfit 

beta 

between 

bevelled 

bibliography 

biconvex 

bi.-^'urcation 

biharmonic 

billoT^ring 

bimetallic 

bimolecular 

binary 

binding 

biot 

biplane 

birnbaum 

bisector 

bistable 

black 

blade 
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03 

0!l 

00 

02 

00 

00 

05 

00 

03 

02 

01 

OS 

01 

00 

00 

01 

27 

06 

01 

03 
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Question 79 What are the details of the rigourous kinetic 
theory of gases (Chapman-Enskog Theory)? 

Index terms: Chapman Enskog gas kinetic 

theory 



Question 100 How much is known about boundary layer flows along 
noncircular cylinders? 

Index terms; boundary cylinder flow layer 
noncircular about 



Question 116 How significant is the possible pressure of a 
dissociated free stream with respect to the 
realization of hypersonic simulation in high 
enthalpy wind tunnels? 

Index terms: dissociated enthalpy free high 

hypersonic pressure simulation 
stream tunnel wind 



Question 118 Do the discrepancies among current analyses of the 
vorticity effect on stagnation-point heat transfer 
result primarily from the differences in the 
viscosity-temperature law assumed? 

Index terms: analysis assumed current difference 

effect heat law point result 
stagnation temperature transfer 
viscosity vorticity 



Question 119 How far can rne trust the linear viscosity 
temperature solution assumed in some of the 
analyses of hypersonic shock layer at low Reynolds 
number? 

Index terms; analysis assumed far hypersonic 

layer linear low number one 
Reynolds shock solution temperature 

viscosity 



Question 121 Has anyone explained the kink in the surge line 
of a multistage axial? 

Index terms: axial compressor line multistage 

surge kink 
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Question 122 Have any aerodynamic derivatives been measured 
at hypersonic Mach numbers and comparison been 
made with theoretical work? 

Index terms: comparison derivative hypersonic 

Mach measured number theoretical 
work 



Question 123 Are the methods of measuring aerodynamic derivatives 
which could be adopted for use in short running 
time facilities? 

Index terms: aerodynamic derivative facility 

measuring method running short 
time use 



Question 126 What are wind-tunnel corrections for a two-dimensional 
aerofoil mounted off-center in a tunnel? 

Index terms: aerofoil correction mounted 

tunnel twodimensional wind 



Question 139 What is the present state of the theory of 
quasi-conical flows? 

Index terms: flow quasiconical state 

theory 



Question 132 What parameters can seriously influence natural 
transition from laminar to turbulent flow on 
a model in a wind tunnel? 

Index terms: flow influence laminar model 

natural parameter transition 
tunnel turbulent wind 



Question 136 How does a satellite orbit contract under the 
action of air drag in an atmosphere in which 
the scale height varies with altitude? 

Index terms: action air altitude atmosphere 

drag height orbit satellite 
scale under 



Question 137 How is the flow at transonic speeds about a 
delta wing different from that on a closely- 
related tapered sweptback wing? 

Index terms: closely delta different 

flow speed sweptback transonic 
wing tapered 
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Question 141 Can methane-air combustion product be used as 
a hypersonic test medium and predict, within 
experimental accuracies, the results obtained 
in air? 

Index terms: accuracy air combustion 

experimental hypersonic medium 
methane product result test 

Question 145 Has anyone investigated the unsteady lift 

distributions on finite wings in subsonic flow? 

Index terms: distribution finite flow 

lift subsonic unsteady wing 

Question 146 What information is available for dynamic response 
of airplanes to gusts or blasts in the subsonic 
regime? 

Index terms: aeroplane blast dynamic gust 

regime response subsonic 



Question 147 Will forward or apex located controls be effective 
at low subsonic speeds and how do they compare with 
conventional trailing-edge flaps? 

Index terms: apex control conventional edge 

effective flap forward located 
low speed subsonic trailing 

Question 148 Given that an uncontrolled vehicle will tumble as 
it enters an atmosphere, is it possible to predict 
when and how it will stop tumbling and its subsequent 
motion? 

Index terms: atmosphere motion tumble 

tumb 1 ing veh i c le 



Question 167 It is not likely that the airforces on a wing of a 
general planform oscillating in transonic flow 
can be determined by purely analytical methods. 

Is it possible to determine the airforces on a 
single particular planform, such as the rectuangular 
one by such methods? 



Question 170 Is there any information on how the addition of a 
"boat-tail** affects the normal force on the body 
of various angles of incidence? 

Index terms: addition angle boattail body 

force incidence normal 
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Question 181 


Has any work been done on determining the nature of 
compressible viscous flow in a straight channel? 

Index terms: channel compressible flow 

nature straight viscous work 


Question 182 


In what areas, other than low density wind tunnel 
flows, is viscous compressible flow in slender 
channels a problem? 

Index terms: area channel compressible density 

flow low problem slender tunnel 

viscous wind 


Question 189 


Has anyone programmed a pump design method for a 
high-speed digital computer? 

Index terms: computer design digital hig.^ 

method programmed pump speed 


Question 190 


Has anyone derived simplified pump design equations 
from the fundamental three-dimensional equations for 
incompressible nonviscous flow? 

Index terms: design equation flow fundamental 

incompressible nonviscous pump 

thr eed imens i onal 


Question 223 


What is the magnitude of second-order wing-body 
interference, at high supersonic Mach number? 

Index terms: body high interference Mach 

magnitude number order second 

supersonic wing 


Question 224 


What is the best theoretical method for calculating 
pressure on the surface of a wing alone? 

Index terms: alone calculating method pressure 

surface theoretical wing 


Question 225 


How can the effect of the boundary- layer on wing 
pressure be calculated, and what is its magnitude? 

Index terms: boundary calculated effect layer 

magnitude pressure wing 


Question 226 


How should the Navier-Stokes difference equations 
be solved? 

Index terms: difference equation navier stokes 
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Question 227 Which iterative method for solving linear difference 
equations is most rapidly convergent? 

Index terms ; convergent difference elliptic 

equation iterative linear method 

Question 230 Technical report on measurement of ablation during 
flight 



Index terms; ablation during flight measurement 

Question 250 What determines the onset of shock-induced boundary- 
layer separation? 

Index terms; boundary induced layer onset 

separation shock 

Question 261 Solution of the Blaslus problem with threepoint 
boundary conditions. 

Index terms; Blasius boundary condition 
problem solution threepoint 

Question 264 References on Lyapunov/s method on the stability 
of linear differential equations with periodic 
coefficients. 

I 

Index terms; coefficient differential equation 

linear Lyapunov method periodic 

re f erence s tabl 1 1 ty 



Question 266 Work on flow in channels at low Reynolds number* 



Index terms; channel flow low number 
Reynolds work 

Question 268 What mode of stalling can be expected for each 
stage of an axial compressor? 

Index terms; axial compressor mode stage 
stalling 

Question 269 Has a criterion been established for determining 
the axial compressor choking line? 

Index terms: axial choking compressor 

criterion line 
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Questivon 272 Has a theory of quasi-confcal flows been 

developed, in supersonic linearized theory, for 
which the upwash distribution on the lifting sur- 
face, apart from being a homogeneous function in the 
cocrdinate, is permitted to have a quite general 
function? 



Index terms ; coordinate developed distribution 
flow form function general 
homogeneous lifting linearized 
quasiconical supersonic surface 
theory upwash 

Question 273 How does scale height vary with altitude in an 
atmosphere? 

Index terms: altitude atmosphere height scale 

Question 274 Jet interference with supersonic flows -- 
theoretical papers. 



Index terms: flow interference jet paper 

supersonic theoretical 

Question 317 Has anyone investigated theoretically whether surface 
flexibility ran stabilize a laminar boundary layer? 

Index terms: boundary flexibility laminar 

layer stabilize surface 

Question 323 How do large changes in new mass ratio quantitatively 
affect wing-flutter? 

Index terms : boundary change flutter large 

mass ratio wing 

Question 360 In practice, how close to reality are the assumptions 
that the flow in a hypersonic shock tube using 
nitrogen is non-viscous and in thermodynamic 
equilibrium? 



Index terms : assumption close equilibrium 

flow hypersonic nitrogen non- 
viscous shock thermodynamic 
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Question Number of Relevant Documents 

Relevant 
Documents 



79 
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1302 


1436 


1437 








100 


4 


1785 


1786 


1787 


1788 






116 


6 


1317 


1574 


1575 


1576 


1578 


1656 
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1324 
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1667 


1670 
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6 


1324 
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1666 
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1713 
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1596 


1597 
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1683 
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2001 


2379 






167 
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1919 


1920 


1921 






170 
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1360 


1605 










181 
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1966 


1967 
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Documents 



Relevant Documents 



182 
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1964 


1965 


1967 


1968 








189 
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1985 


1990 
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1984 
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1986 


1987 


1988 


1989 


1990 


223 


2 


2074 


2075 












224 
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226 
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227 
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2088 
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2100 
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2364 






2367 














261 
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1476 








264 
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1451 












266 
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1964 
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269 
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1591 
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138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 

159 

160 
161 
162 

163 

164 

165 

; 166 
O 167 

ERIC 



TABLE 

QUARTER IffiLETIONS 



Cum. Freq. 


Term 


3468 


POTENTIAL 


3481 


BUOE 


3^+94 


HEATING 


3507 


FLIGHT 


3520 


FRICTION 


3533 


mi 


35^46 


SfOTTBACK 


3559 


SMALL 


3572 


UPPER 


3585 


TRANSITION 


3598 


REVOLUTION 


3611 


SATELLITE 


3623 


SPAMSE 


3635 


THREEDIMENSIONAL 


3647 


THRUST 


3659 


TIME 


3671 


VORTICITY 


3683 


LOADING 


3695 


LOAD 


3707 


GRADIENT 


3719 


AXIAL 


3731 


IffiLTA 


3743 


DATA 


3755 


CHORIWISE 


3767 


CONSTANT 


3778 


CONE 


3789 


COMPRESSOR 


3800 


CURVATURE 


2811 


FLUTTER 


3822 


DISPLACEMENT 


3833 


EXHAUSTING 


3844 


LOCAL 


3855 


LONGITUDINAL 


3866 


APPROXIMATION 


3877 


TRAILING 


3888 


TUBE 


3898 


REDUCTION 


3908 


TIP 


3918 


WEIGHT 


3928 


WEDGE 


3938 


SLOPE 


3948 


ZERO 


3958 


SOLED 


3968 


SSfffiPT 


3978 


REACTION 


3988 


SPAN 


3998 


RECTANGULAR 


4008 


PATTERN 


4018 


LENGTH 



354 



THIRD 

Freq. 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 



348 



TABLE, —continued 



THIRD QUARTER DELETIONS 



Class 


Preq. 


Cura. Freq. 


Terra 


168 


10 


4028 


MAHMUl-! 


169 


10 


4038 


I-5ATERIAL 


170 


10 


4048 


MASS 


171 


10 


4058 


OSCILLATION 


172 


10 


4068 


OSCILLATING 


173 


10 


4078 


ORBIT 


174 


10 


4088 


EXPERI-f-lENT 


175 


10 


4098 


PERFORMANCE 


176 


9 


4107 


DERIVATIVE 


177 


9 


4116 


CONICAL 


178 


9 


4125 


POSITION 


179 


9 


4134 


ENTHALPY 


180 


9 


4143 


D04NVASH 


181 


9 


4152 


FREQUENCY 


182 


9 


4l6l 


RANGE 


183 


9 


4170 


OSCIILATCRY 


184 


9 


4179 


LIFTING 


185 


9 


4188 


IDDE 


186 


9 


4197 


APPROmiATE 


187 


9 


4206 


REUTEON 


188 


9 


4215 


WAKE 


189 


9 


4224 


TRANSIENT 


190 


9 


4233 


THEN 


191 


9 


4242 


RISE 


192 


9 


4251 


UNSTEADY 


193 


9 


4260 


REGION 


19': 


Q 


'■•269 


STREAl'IINE 


195 


8 


4277 


STIFFNESS 


196 


8 


4285 


SPECIFIC 


197 


8 


4293 


SYSTEI-5 


198 


8 


4301 


SLENDER 


199 


8 


4309 


VERTECAL 


200 


8 


4317 


SIZE 


201 


8 


4325 


ARBITRARY 


2 q 2 


8 


4333 


PSRIGREE 


203 


8 


43/il 


FIXING 


204 


8 


4349 


FIXED 


205 


8 


4357 


LINE 


206 


8 


4365 


PLANE 


207 


8 


4373 


RADIATION 


208 


8 


4381 


PROBLEM 


209 


8 


4389 


NOSED 


210 


8 


4397 


INDECAL 


211 


8 


4405 


INLET 


212 


8 


4413 


PLANFQFJ'i 


213 


8 


4i).21 


INTEGRAL 


214 


8 


4429 


FORi'JARD 


215 


8 


'4437 


Da-INSTREAM 




355 



349 



TABLE. — continued (3) 
THIRD QUARTER DELETIONS 






Class 


Freq. 


Cum. FVeq( 


216 


8 




217 


8 


4^+53 


218 


8 


4^+61 


219 


8 


4469 


220 


8 


4477 


221 


8 


i^485 


222 


8 


4493 


223 


8 


4301 


224 


8 


4509 


223 


7 


4516 


226 


7 


4523 


227 


7 


4530 


223 


7 


4537 


229 


7 


454 ^ 1 . 


230 


7 


4531 


231 


7 


4558 


232 


7 


4565 


233 


7 


4572 


234 


7 


4379 


235 


7 


4586 


236 


7 


4593 


237 


7 


4600 


238 


7 


4607 


239 


7 


4614 


240 


7 


4621 


241 


7 


■’+628 


242 


7 


4633 


243 


7 


4642 


244 


7 


4649 


243 


7 


46 56 


246 


7 


4663 


247 


6 


4669 


248 


6 


4673 


249 


6 


4681 


250 


6 


4687 


251 


6 


^+693 


252 


6 


4699 


233 


6 


4703 


25*4 


6 


4711 


235 


6 


4717 


256 


6 


4723 


257 


6 


4729 


258 


6 


4733 


259 


6 


4741 


260 


6 


4747 


261 


6 


4753 


262 


6 


4759 


263 


6 


4763 


264 


6 


4771 



Term 

EDGED 

EHT 

EXHAUST 

HEIGHT 

CONDUCTION 

DTAl-ETER 

CURVE 

CHANGE 

BENDING 

PITCH 

CHORD 

DIFFERENTIAL 

CONFIGURATION 

RELAXAHON 

DISSOCIATION 

DISTURBANCE 

FOR^^ 

RESULT 

PROPERTY 

PARALLEL 

LINEAR 

SY1#ETRICAL 

ROTATING 

SWEEP 

STRESS 

SPHERICAL 

SCALE 

UNIFORM 

SIMULATION 

UPSTREAM 

THERI'IAL 

THICK 

VARIABLE 

SHEAR 

TYPE 

STALL 

STANDOFF 

VIBRATION 

PERIOD 

LINEARIZED 

PARABOLIC 

PARA!-1ETER 

ONSET 

REFLECnON 

INFINITE 

RIGID 

INVISCID 

MOIfflNTUI"! 

PLANETARY 



350 

TABLEt — contAnued (^) 



THIRD QUARTER DELETIONS 



Class 


Freq. 


Cum* Freq* 


Term 


265 


6 


4777 


GRAVin 


266 


6 


4783 


FLAP 


26? 


6 


4789 


ECCENTRICITY 


268 


6 


4795 


EARTH 


269 


6 


4801 


DYNAl'ilC 


270 


6 


4807 


ENERGY 


271 


6 


4813 


HALF 


272 


6 


4819 


GUST 


273 


6 


4825 


II-PELLER 


274 


6 


4831 


CONTRACTION 


275 


6 


4837 


ROTATIONAL 


276 


6 


4843 


30'.' 


277 


6 


4849 


BLUNTED 


278 


6 


4855 


BEHIND 


279 


6 


4861 


BASE 


280 


6 


m? 


AnSYI'l'ETRIC 


281 


6 


4873 


AXIS 


282 


6 


4879 


ASB2PT0TIC 


283 


6 


4885 


ATl'iOSPHERIC 


28'i- 


6 


4891 


ARC 


28''. 


6 


4897 


AFTERBODY 


286 


6 


4903 


ANALYTICAL 


287 


5 


/J .908 


PATH 


288 


5 


4913 


ACCELERATION 


289 


5 


4918 


BLO/JING 


290 


5 


4923 


BLOCKAGE 


291 


c 

V 


4928 


BUFFETING 


292 


5 


4933 


CHOKING 


293 


5 


4938 


CHOKED 


29'!- 


5 


4943 


CHANNEL 


295 


5 


4948 


CENTRIFUGAL 


296 


5 


4953 


CONTINUOUS 


297 


5 


4958 


PRAMDTL 


298 


5 


4963 


COI-iPRESSION 


299 




4968 


COMPLETE 


300 


5 


4973 


CORRECTION 


301 


5 


4978 


DIFFERENCE 


302 


5 


4983 


DSTACm-aENT 


303 


5 


4988 


CROSS 


30'i 


5 


' 1.992 


DEPEMDEt.T 


305 


5 


'.'•998 


DEFLECTION 


306 


5 


5003 


PHOTOGRAPH 


307 


5 


5008 


IDEAL 


308 


5 


5013 


HOT 


309 


5 


5018 


PERTURBATION 


310 


5 


5023 


GROTTH 


311 


5 


5028 


ENGINE 


312 


5 


5033 


RELATIVE 



O 

ERIC 
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351 



TABLE. — continued (5) 
THIRD QUARTER DELETIONS 



Class 



Freq. Cum. Freq. 



313 

314 

315 

316 

317 

318 

319 

320 

321 

322 

323 
32-4 

325 

326 

327 

328 
32.9 

330 

331 

332 

333 

33 ':- 

335 

336 

337 

338 

339 
T-0 
3''1 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



5038 

5043 

5048 

5053 

5058 

5063 

5068 

5073 

5078 

5083 

5088 

5093 

5098 

5103 

5108 

5113 

5118 

5123 

5128 

5133 

5138 

5143 

5148 

5153 

5158 

5163 

5168 

5173 

5178 




358 



Term 

EUSTIC 

DIVERGENT 

GLIDER 

GENERALI : 3 SD 

REAR 

ENTRY 

EXTERML 

MOLECUUR 

MODIFIED 

NElr/TONIAN 

ISENTROPIC 

INVESTIGATION 

ONEDIl'ffiNSlONAL 

OXYGEN 

NOKVISCOU 3 

NONLINEAR 

NONEQUILIBRIUI-; 

LONG 

MAIN 

VISCOSITY 

WORKING 

SURVEY 

STUDY 

SPACE 

SUP 

STRUCTURE 

STRENGTH 

TESTING 

SINGLE 



352 



Class 

y^2 

3’^■3 

344 

345 

346 

347 

3ii6 

349 

350 

351 

352 

353 

3^4 

355 

356 

357 

358 

359 

360 

36 1 

362 

363 

364 

365 

366 

367 

368 

369 

370 

371 

372 

373 

374 

375 

376 

377 

378 

379 

380 

381 
382 
383 
38'i- 

385 

386 

387 

388 

389 

ERIC 



TABLE 

QUARTER DELETIONS 



it Freq# 


Terra 


5183 


SINKING 


5188 


SIMULATED 


5192 


WO 


5196 


TRIANGULAR 


5200 


TURBOJET 


5204 


VANE 


5208 


TRANSITIONAL 


5212 


TRAJECTORY 


5216 


THROAT 


5220 


STATE 


5224 


IJIDTH 


5228 


TAIL 


5232 


TABLE 


5236 


SUCTION 


5240 


SURGE 


52 W 1 - 


VECTOR 


<=>?M 


STAGE 


5252 


SHROUD 


5256 


SIDESLIP 


5260 


NONLIFTING 


5264 


PARTICLE 


5268 


PANEL 


5272 


PROBE 


5276 


OIL 


5280 


ORBITAL 


5284 


INITIAL 


5288 


INTEGRATION 


5292 


NITROGEN 


5296 


NOISE 


5300 


MXTURE 


5304 


NACELLE 


5308 


MULTIPLE 


5312 


FACTOR 


5316 


FACE 


5320 


ENTROPY 


5324 


PHASE 


5328 


ESTIMATION 


5332 


RESPOIBE 


5336 


RAHIAL 


5340 


FUSELAGE 


53;.i4 


FULLY 


5348 


FROZEN 


5352 


FORJ4ULA 


5356 


FOREBODY 


5360 


FRACTION 


5364 


DIVERGENCE 


5368 


PENETRATION 


5372 


Pa^TER 


5376 


DISSOCIATING 



359 



FOURTH 

Freq. 

5 

5 

4 

4 

4 

4 

4 

4 

4 

4 

4 

ij, 

4 

4 

4 

4 

4 

4 

4 

4 

4 

h 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

i! 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 



353 

TABLEi — continued (2) 
FOURTH QUARTER DELETIONS 




Class 


Freq. 


Cunii Freq# 


Terra 


391 


4 


5380 


SEPARATED 


392 


4 


5384 


EFFICIENCY 


393 


4 


5388 


EDDY 


394 


4 


5392 


SOLAR 


395 


4 


5?96 


ROLLING 


396 


4 


5400 


REAL 


397 


r 


5404 


DRIVING 


398 


4 


5408 


HEAD 


399 


4 


5412 


HUB 


400 


4 


5416 


HORSESHOE 


401 


4 


5420 


ROTATION 


402 


4 


5424 


HEUUl'^ 


403 


4 


5428 


SEGMENT 


404 


4 


5432 


PROPULSIVE 


405 


4 


5/^36 


DECELERATION 


406 


4 


5440 


DETACHED 


407 


4 


5444 


CORNER 


408 


4 


5ii48 


COOLING 


409 


4 


54 <^2 


CO^^PRESSIBILITY 


410 


4 


5456 


CONVECTION 


411 


4 


5460 


CONVERGENT 


412 


4 


5464 


CONVECTIVE 


413 


4 


<7+68 


CONDUCTinTY 


414 


4 


5^f-72 


SCHIIEREN 


41 


4 


5476 


CHEMCAL 


416 


4 


5ii-80 


PERCENT 


417 


4 


5484 


CAI'BER 


418 


4 


5488 


BLA5IUS 


419 


4 


5492 


ATTACHED 


420 


4 


5496 


BEHAVIOR 


421 


4 


5500 


ABUTION 


422 


4 


5504 


AIRCRAFT 


423 


3 


5507 


ANALYTIC 


424 


3 


5510 


AMBIENT 


425 


3 


5513 


ADIABATIC 


426 


3 


5516 


RECOVERY 


427 


3 


5519 


SPUTNIK 


428 


3 


5522 


AREA 


429 


3 


5525 


BUDING 


430 


3 


5528 


BLa-/ 


431 


3 


5531 


BLUNTNESS 


432 


3 


553^ 


STALLING 


433 


3 


5537 


BALANCE 


434 


3 


5540 


ROUGHNESS 


435 


3 


^543 


BOATTAIL 


436 


3 


55i^6 


CHEMECALLY 


437 


3 


5549 


REQUIREMENT 


438 


3 


5552 


CLOSED 



360 



TABLE. — continued (3) 
FOURTH QUARTER DELETIONS 



Class 

>09 
•'+40 
4.’'-l 
■4' 1-2 

!tll7 
44/'. 
44-^ 
).ai6 
4-47 
' 41-8 
1,119 
4 SO 
4 Si 

452 

453 
i.!-S 4 

455 

456 

457 

458 

459 

460 

461 
•462 
46? 
'I-64- 
46 s 

466 

467 
'+68 

469 

470 

471 

472 

473 

474 

475 

476 

477 

478 

479 

480 

481 

482 
482 
484 
48 s 

486 

487 

O 488 

ERIC 



Cum. Freq. 


Term 


5555 


STATIONARY 


5558 


COIffllNED 


5561 


CIRCULATION 


5564 


PLOTTING 


5567 


RAYLEIGH 


5570 


CONSTRUCTION 


5573 


SPHERE 


5576 


CONVERGING 


5579 


CONVERGENCE 


5582 


ROTOR 


5585 


CONTOUR 


5588 


STOKES 


5591 


CONTI NUUI-I 


5594 


COiSUSTION 


5597 


STRAIGHT 


S600 


COMPUTER 


s 603 


COORDINATE 


s606 


CONVEX 


5609 


DEVELOPIENT 


5612 


DECAY 


561s 


STREAH-.TISE 


56I8 


CYLIUmiCAL 


5621 


DISCONTINUITY 


S 624 


DISCHARGE 


5627 


DIHEDRAL 


5630 


STRIP 


‘5633 


STRONG 


5636 


STRUCTURAL 


5639 


HISTORY 


S 642 


HEMISPHERICAL 


S 64 S 


HEMSPHERE 


S 648 


SUBLIMATION 


5651 


IMPACT 


5654 


SOURCE 


5657 


SUCCESSIVE 


5660 


HYDROGEN 


5663 


SEI'IIIMNITE 


5666 


SEl'IIMAJOR 


5669 


INCREMENTAL 


5672 


II-IPULSE 


5675 


SOUND 


5678 


DOUBLE 


S 681 


DURATION 


S68^+ 


EFFECTIVE 


5687 


ELECTRICAL 


5690 


EUSTICITY 


5693 


DISSOCIATED 


^96 


ROOT 


5699 


S'/EEPBACK 


5702 


SLOTTED 



Freqi 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

2 

3 

3 

3 

3 

2 

'2 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 
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TABLE. — continued ( 4 ) 



Class 

489 

490 

491 

492 

493 

494 

495 

496 

497 

498 

499 

500 

501 
<502 
503 

<504 

505 

506 

507 

508 

509 

510 

511 

512 

513 

514 

515 

516 

517 

518 

519 

520 

521 

522 

523 

524 

525 

526 

527 

528 

529 

530 

531 

532 

533 

534 

535 

536 



ERIC 



QUARTER DELETIONS 



[• Freq. 


Term 


5705 


SLOT 


5708 


FORMATION 


5711 


FREEDOI^ 


5714 


FRICTIONAL 


5717 


GENERATORS 


5720 


SERIES 


5723 


GRAPHITE 


5726 


GEOIffiTRY 


5729 


FLUX 


5732 


FLEXIBILITY 


5735 


EXCITATION 


5738 


EXPLORER 


5741 


SINUSODIAL 


574^+ 


MOVING 


5747 


TANGENTIAL 


5750 


NACA 


5753 


TAPER 


5756 


RECOrSINATION 


5759 


I'USSILE 


5762 


NODE 


5765 


NEARLY 


5768 


NAVIERSTOKE.S 


5771 


INTERl'EDIATE 


5774 


INFLUENCE 


5777 


REGIlffi 


5780 


IONIZED 


5783 


IONIZATION 


5786 


PISTON 


5789 


ITERATIVE 


5792 


REATTACHMENT 


5795 


OPERATION 


5798 


PHENOMENA 


5801 


OBUTE 


5804 


PARTIAL 


5807 


PART 


5810 


THERMODYNAMIC 


“;813 


OUTLET 


5816 


OUTBOARD 


5819 


REFLECTED 


5822 


LOGARITHI^EC 


5825 


LOSS 


5828 


RAE 


5831 


REVERSAL 


5834 


THREEPOINT 


5837 


LIMT 


5840 


LIGHT 


5843 


LEEtiTARD 


58^16 


RADIUS 



FOURTH 

Freq. 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 



362 



356 



TABLE. — continued (5) 
FOURTH QUARTER DELETIONS 



O 

ERIC 



Clas.'. 


Freq. 


Cum* Freq 


53? 


3 


5849 


53r? 


3 


5852 


■'39 


3 


5855 


5'-iO 


3 


5858 




3 


5861 


5'-'-2 


3 


5864 


5'+3 


y 


5867 


57i-7i- 


3 


5870 


^<•5 


3 


5873 




3 


5876 


547 


2 


5878 


S'!-8 


2 


5880 


549 


2 


5882 


550 


2 


5884 


551 


2 


5886 


552 


2 . 


5888 


553 


2 


5890 


557,'. 


2 


5892 


555 


2 


5894 


556 


2 


5896 


557 


2 


5898 


558 


2 


5900 


559 


2 


5902 


560 


2 


5904 


561 


2 


5906 


562 


2 


5908 


563 


2 


5910 


564 


2 


5912 


565 


2 


5914 


566 


2 


5916 


567 


2 


5918 


568 


2 


5920 


569 


2 


5922 


570 


2 


'5924 


571 


2 


5926 


572 


2 


5928 


573 


2 


5930 


574 


2 


5932 


575 


2 


593/.U 


576 


2 


5936 


577 


2 


5938 


578 


2 


5940 


579 


2 


5942 


580 


2 


59Wj. 


581 


2 


5946 


582 


2 


5948 


583 


2 


5950 


5a4 


2 


5952 


585 


2 


59,5^+ 



Term 

LARGE 

UTITUDE 

LATERAL 

W 

V7IRE 

TORSIONAL 

TRAIBPORT 

VALUE 

VANISHING 

T'TIST 

LIEBl-AN 

PERFECT 

THROUGH 

RECIPROCAL 

LIFETIME 

PROPELLER 

LIQUID 

MATCHING 

SIDS 

MNOEUVRE 

RANDOM 

TRA^B FORMATION 

MjARGIK 

PROPELLANTS 

RADIAnVE 

TRANSUTION 

PERCENTAGE 

LYAPUNOV 

PIVOTAL 

NONSTATIONARY 

NONSTEADY 

THICKENING 

OUTER 

RETROROCKET 

TRIM 

SHIELD 

OBSERVATION 

TUl'SUNG 

OBUQUE 

OGIVE 

SHEET 

RICHARDSON 

ONE 

OPEN 

RAREFIED 

ORDER 

TV-.TSTED 

KNEE 

HRSCHHOFF 



363 



357 

TABLEi — continued (6) 



FOURTH QU/LRTER DELETIONS 





Class 


Freqt 


Cumt Freqt 


Term 




586 


2 


5956 


KINETIC 




587 


2 


5958 


KILOI'ETRE 




588 


2 


5960 


KERNEL 




589 


2 


5962 


KELVIN 




590 


2 


5964 


TERMKAL 




591 


2 


5966 


INVERSION 




592 


2 


5968 


INTERSECTING 




593 


2 


5970 


TENSION 




59 'J- 


2 


5972 


INTERNAL 




595 


2 


5974 


INTERFACE 




596 


2 


5976 


INfHR 




597 


2 


5978 


TEFLON 




598 


2 


5980 


NEAR 




599 


2 


5982 


NAVIER 




600 


2 


5984 


TECmttQUS 




601 


2 


5986 


NET 




602 


2 


5988 


SINK 




603 


2 


5990 


TOBWEPT 




60't 


2 


5992 


KODAL 




605 


2 


5994 


TAPERED 




606 


2 


5996 


NATURAL 




607 


2 


5998 


NARRG'J 




608 


2 


6000 


IIULTISTAGE 




609 


2 


600 ? 


UP'IASH 




610 


2 


6004 


MOVEl^ENT 




611 


2 


6006 


SHADa/GRAPH 




612 


2 


6008 


MOLECULE 




613 


2 


6010 


PIVOT 




61 ' 1. 


2 


6012 


I-ECHANICS 




615 


2 


601 i'- 


MEAN 




616 


2 


6016 


■ I-EIER 




617 


2 


6018 


MERIDIONAL 




618 


2 


6020 


I'ERGED 




619 


2 


6022 


PRir-IARY 




620 


2 


6024 


EXCESS 


‘ 


621 


2 


6026 


SLAB 


i’ 

i 


622 


2 


6028 


ETHYLENE 


?• 


623 


2 


6030 


ESCAPE 


I 


624 


2 .. 


6032 


EQUIVALENT 


1 : 


625 


2 '■ 


6034 


FACED 


1 


626 


2 


6036 


EXTENSION 




627 


2 


6038 


EXPOSED 


1 


628 


2 


6040 


REACTING 


629 


2 


60 ^ 1-2 


FIRST 


0- 


630 


2 


6OI1I'. 


FILAMENT 




631 


2 


6046 


VISCID 


■ '‘v 


632 


2 


6048 


FLAl'^ 




633 


2 


6050 


FIXED 


#• O 

lERiC 








364 











358 

TABLE. — continued (?) 
FOURTH QUARTER DELETIONS 



( ■ 

I 

i 

t 



I O 

lERlC 



Class 


Freq. 


Cura. Freq. 


Terra 


63^'- 


2 


6052 


FLEXIBLE 


635 


2 


6054 


QUASI 


636 


2 


6056 


ROLL 


637 


2 


6058 


VORTICES- 


638 


2 


6060 


PHOTOGRAPHIC 


639 


2 


6062 


PENETRATING 


640 


2 


606^1 


FLUCTUATION 


64.1 


2 


6066 


GUDE 


642 


2 


6068 


TAB 


643 


2 


6070 


GENERATED 




2 


6072 


FUEL 


645 


2 


6074 


FRONT 


646 


2 


6076 


FREEZING 


647 


2 


6078 


SmiETRIC 


6i.'.8 


2 


6080 


S’-.-EPTFORJ/URD 


649 


2 


6082 


FOURIER 


650 


2 


60 m 


FORCED 


651 


2 


6086 


FOLDING 


652 


2 


6088 


’TDE 


653 


2 


6090 


ENTERING 


654 


2 


6092 


EITSSIVITY 


655 , 


2 


6094 


I'TNWARD 


656 


2 


6096 


ELLIPTICAL 


657 


2 


6098 


’TNGED 


658 


2 


6100 


ELEI'SSNT 


659 


2 


6102 


DISTRIBUTED 


660 


2 


610^'. 


SLQ-rLY 


661 


2 


6106 


DIVERGING 


662 


2 


6108 


DIURNAL 


663 


2 


6110 


YA!:5ED 


664 


2 


6112 


QUASICONICAL 


665 


2 


6114 


DESK 


666 


2 


6116 


ZOl^E 


667 


2 


6118 


ELECTRICALLY 


668 


2 


6120 


SODimi 


669 


2 


6122 


DOUBLET 


670 


2 


6124 


SUPPORT 


671 


2 


6126 


PROCESS 


672 


2 


6128 


INCUNATION 


673 


2 


6130 


INCIPIENT 


67 ''- 


2 


6132 


INCONEL 


675 


2 


6134 


SUPERPOSITION 


676 


2 


6136 


GROUND 


677 


2 


6138 


GROSS 


678 


2 


61U0 


HEATED 


679 


2 


6 lh 2 


GUIDE 


680 


2 




ROTARY 


681 


2 


6146 


HARMDNICALLY 


682 


2 


6148 


HYDRODYNAITC 



365 



359 

TABLE. — continued (8) 



Class 

683 

68 '-. 

685 

686 
68 ? 
688 

689 

690 

691 

692 

693 

69 ^ 1 - 

695 

696 

697 

698 

699 

700 

701 

702 
703 

7b'i 

705 

706 

707 

708 

709 

710 

711 

712 

713 

71'' 

713 

716 

717 

718 

719 

720 

721 

722 

723 

72^!- 

723 

726 

727 

728 

729 

730 

er|c 



QUARTER DELETIONS 
Cura. Freq. 


Term 


6150 


SPACING 


6152 


HYSTERESIS 


6152; 


HYPERVELC-CITY 


6156 


HIGHLY 


6158 


HOl'iOGENEOlB 


6160 


DIFFERENTIALLY 


6162 


DIRECTICJIAL 


6164 


PROGRESSIVE 


6166 


CURVED 


6168 


DEFLECTED 


6170 


removal 


6172 


DEVIATION 


6174 


SECTTOML 


6176 


DEVELOPED 


6178 


RESONANCE 


6180 


SPECTRA 


6182 


SECONDCRDER 


6184 


COPPER 


6186 


CRITICAL 


6188 


COUETTE 


6190 


CONCAVE 


6192 


COKPOf.ENT 


6194 


SECOND 


6196 


SEASON 


6198 


PUI'P 


6200 


SPHERICALLY 


6202 


COI'BINATION 


6204 


COLLISION 


6206 


COLD 


6208 


PREa^HIRL 


6210 


CHAIBER 


621? 


PERIODIC 


62L-1 


STARTING 


6216 


BU1"1P 


6218 


BUZZ 


6220 


BURNED 


6222 


CASCADE 


6224 


CARBON 


6226 


BALLISTIC 


6228 


RO-' 


6230 


PROPULSION 


6232 


AUGI-ENTATION 


623''- 


ATTENUATION 


6236 


ATTACHt-ffiNT 


6238 


AVERAGE 


6240 


STALLED 


6242 


BLO/JDa-VN 


624^+ 


BLEED 



FOURTH 

Freq# 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



366 



360 

TABLE* — continued (9) 
FOURTH QUARTER DELETIOMS 



er|c 



Class 


Freq. 


Cum. Freq. 


Term 


731 


2 


6246 


ATOIILC 


732 


2 


6248 


PIATES 


733 


2 


6250 


POHLHAUSEN 


734 


2 


6232 


ARTIFICIAL 


733 


2 


625^1- 


BEAK 


736 


2 


6236 


BETv'EESK 


737 


2 


6238 


APPLIED 


738 


2 


6260 


APOGEE 


739 


2 


6262 


ADJACENT 


7''0 


2 


6264'- 


AHEAD 


741 


2 


6266 


PRESCRIBED 


742 


2 


6268 


PERPENDICULARLY 


7':-3 


2 


6270 


STACKING 


7’ I-'! 


2 


6272 


Ai^omiT 


743 


2 


6274'- 


STABLE 


74-6 


1 


6273 


SAIL 


747 


1 


6276 


POLY>3R 


74-8 


1 


6277 


ANALOGY 


749 


1 


6278 


Al-TLITUDE 


750 


1 


6279 


ALUI'OINUH 


731 


1 


6280 


PHOTORECORDING 


732 


1 


6281 


AIRSPEED 


753 


1 


6282 


AILERON 


73^.'- 


1 


6283 


PRECIPITATION 


755 


1 


628-4 


ALTERNATING 


756 


1 


6283 


ALLOY 


757 


1 


6286 


ALU-IOVABLE 


758 


1 


6287 


AEROELASTICITY 


739 


1 


6288 


ADVERSE 


760 


1 


6289 


SAFETY 


761 


1 


6290 


AER0THERl'10DYNAI“n:c 


762 


1 


6291 


AEROPLANE 


763 


1 


6292 


ADJUSTABLE 


764 


1 


6293 


ADDITION 


765 


1 


6294 


ACTUAL 


766 


1 


6293 


ACTIVITY 


767 


1 


6296 


ACTIVE 


768 


1 


6297 


ACOUSTIC 


769 


1 


6298 


APPARENT 


770 


1 


6299 


APPARATUS 


771 


1 


6300 


APEX 


772 


1 


6301 


APPROACHING 


773 


1 


6302 


APPROACH 


774 


1 


6303 


APPEARANCE 


775 


1 


63CVE 


STAGGER 


776 


1 


6305 


RUNOTNG 


777 


1 


6306 


RECORD 


778 


1 


6307 


RADIATOR 


779 


1 


6308 


ANTISYI^I»]ETRIC 


780 


1 


6309 


ANNULUS 



367 



361 

TABLE. — continued ( 10 ) 
FOURTH QUARTER DELETIONS 




Class 


Freq. 


Cum. Freq. 


781 


1 


6310 


782 


1 


6311 


783 


1 


6312 


78'i- 


1 


6313 


78“; 


1 


631 '^ 


786 


1 


6315 


787 


1 


6316 


788 


1 


6317 


789 


1 


6318 


790 


1 


6319 


791 


1 


6320 


792 


1 


6321 


793 


1 


6322 


79 '! 


1 


6323 


795 


1 


6324 


796 


1 • 


6325 


797 


1 


6326 


798 


1 


6327 


799 


1 


6328 


800 


1 


6329 


801 


1 


6330 


802 


1 


6331 


803 


1 


6332 


80 ! 


1 


6333 

633 't 


805 


1 


806 


1 


6335 


807 


1 


6336 


808 


1 


6337 


809 


1 


6338 


810 


1 


6339 


811 


1 


63 'iO 


812 


1 


63^'-1 


813 


1 


6342 


81 ^. 


1 


6y'-3 


81*5 


1 


6y\H 


816 


1 




817 


1 


63-^ 


818 


1 


6347 


819 


1 


6348 


820 


1 


6349 


821 


1 


6350 


822 


1 


6351 


823 


1 


6352 


82 i|. 


1 


6353 


825 


1 


6354 


826 


1 


6355 


827 


1 


6356 


328 


1 


6357 


829 


1 


6358 



Term 

ANEKOi-lETER 

SQUARE 

ABOVE 

ABOUT 

ABUTING 

SANDPAPER 

ACCURATE 

ACCOM-IODATION 

ACCELERATING 

SALT 

PAST 

PASSING 

PARTITION 

RADIOACTIVE 

PEAKY 

PEAK 

PAY - 

PENDULUIi'' 
PENALTY ' 
PECLET 
ABEL 

ACCELERATED 

ABSORPTION 

ABSOLUTE 

ABRUPT 

BIPLANE 

BIOT 

BENEATH 

ROUTINE 

BEAD 

BASIC 

STAGGERED 

ROUNDINGOFF 

RESEARCH 

BELT 

BANGBANG 

BAND 

BALSA 

BALLOTINI 

ASSOCIATED 

ARRa-/ 

RUDDER 

ATLAS 

A 3 Y 1 # 3 ETRIC 

SPOILER 

RULE 

ATOM 

ARRANGEl'ENTS 

ARGON 



368 



362 

TABLE. — continued ( 11 ) 



FOURTH QUARTER DELETIONS 



Class 


Freq« 


Cum* Preq 


830 


1 


6339 


831 


1 


6360 


832 


1 


6361 


833 


1 


6362 


83/:. 


1 


6363 


835 


1 


6364 


836 


1 


6363 


837 


1 


6366 


83,8 


1 


6367 


839 


1 


6368 


8i\0 


1 


6369 


mi 


1 


6370 


8 'i -2 


1 


6371 


a'4-3 


1 


6372 




1 


6373 


8 /..'S 


1 


6374 


8''-6 


1 


6375 


8 U 7 


1 


6376 


a +8 


1 


6377 


8.’.!.9 


1 


6378 


850 


1 


6379 


831 


1 


6380 


832 


1 


6381 


833 


1 


6382 


83 ii 


1 


6383 


855 


1 


6384 


8 36 


1 


6383 


837 


1 


6386 


838 


1 


6387 


839 


1 


6388 


860 


1 ' 


. 6389 


861 


1 


6390 


862 


1 


6391 


863 


1 


6392 


8m 


1 


6393 


863 


1 


639 '.'- 


866 


1 


6393 


867 


1 


6396 


868 


1 


6397 


869 


1 


6398 


870 


1 


6399 


871 


1 


6400 


872 


1 


6401 


873 


1 


6402 


874 


1 


6403 


875 


1 


6404 


876 


1 


mo 5 


877 


1 


6406 



er|c 



Term 

ARDC 

SATURATION 

STANDARD 

ROUl'lDED 

RESERVOIR 

BLUNTING 

SPLITTER 

PROLATE 

BIRNBAUK 

BLUFF 

BLC^^N 

BLOWER 

AVRO 

AUTOPILOT 

AUTOMATIC 

PICKUP 

ROUTTHURIf/ITZ 

AXIALLY 

BAFFLES 

BACK 

CARBORUimUM 

CAPTURE 

CAPACITY 

CENTIMETRE 

CENTERING 

CARRYING 

SPIN 

PERPENDICULAR 

CAMBERED 

CALCULATING 

CAPACITANCE 

CAPABILITY 

CANTELEVERED 

CANTELEVER 

BURSTING 

BUOYANCY 

BUFFET 

SUCKLING 

SPIKED 

BREATHING 

BRADING 

BOUND 

BOOST 

STATICALLY 

SCHLICHTING 

SPIKE 

CHLORIDE 

CHARGED 



369 



363 

TABLE. — continued (12) 
FOURTH QUARTER DELETIONS 



Class 


Fraq. 


Cum. Freq. 


878 


1 


6407 


879 


1 


6-408 


880 


1 


6409 


881 


1 


6410 


882 


1 


6411 


883 


1 


6412 


88.''. 


1 


6413 


88 


1 


6414 


886 


1 


641 <3 


887 


1 


6+16 


888 


1 


6417 


889 


1 


6418 


890 


1 


6419 


891 


1 


6420 


892 


1 


6421 


893 


1 


6422 


894 


1 


6423 


893 


1 


6424 


896 


1 


6423 


897 


1 


6426 


898 


1 


6427 


899 


1 


6428 


900 


1 


64-29 


901 


1 


6430 


902 


1 


6431 


903 


1 


6432 


904 


1 


61-33 


905 


1 


6i.iy.,. 


906 


1 


643-3 


907 


1 


6«-36 


908 


1 


6437 


909 


1 


6438 


910 


1 


6439 


911 


1 


6440 


912 


1 


647+1 


913 


1 


644-2 


91'!- 


1 


64'+3 


913 


1 


6Wi- 


916 


1 


6i+'+5 


917 


1 


6446 


918 


1 


6447 


919 


1 


6448 


920 


1 


6449 


921 


1 


6450 


922 


i 


6451 


923 


1 


6452 


924 


1 


6+53 


925 


1 


6454 



O 

ERIC 



Term 

SPHEROID 

PRODUCTION 

CBSSATION 

CENTRAL 

SCAVENGING 

SCALING 

RESISTANCE 

CHOKE 

STATOR 

CLASSICAL 

CIRCUIIFERENTIAL 

CIRCUUTORY 

STATISTICAL 

COCURRENT 

CLIPPED 

PROmiTY 

POUND 

CIRCULATING 

CIRCUIT 

STEEL 

RELIABILITY 

CONDUCTING 

STIFFE^ER 

CONSTRAINT 

CONSERVATION 

STEP 

RESOMAOT 

PURE 

CONICALLY 

CONDEJSATION 

CONCENTRIC 

CONCENTRATION 

CONCENTRATED 

CONTINUITY 

CONTINUATION 

CONTACT 

STORES 

STORE 

STOICHIOI'ETRIC 

CONTRACTING 

STING 

SULL 

PLUNGING 

CONTROLLING 

CONTRIBUTION 

SPECTRUM 

SEASONAL 

COl'iPRESSED 



370 



364 



TABLE. — continued (13) 
FOURTH QUARTER DELETIONS 



Class Freq. Cum. Freq. Terra 



926 1 

92? 1 

928 1 

929 1 

930 1 

931 1 

932 1 

93" 1 

934 1 

935 1 

936 1 

937 1 

938 1 

939 1 

940 1 

941 1 

9>iz 1 

943 1 

9^41- 1 

9'! 5 1 

946 1 

947 1 

9’--8 1 

9':-9 1 

950 1 

951 1 

952 1 

953 1 

95/^ 1 

955 1 

956 1 

9‘^7 1 

958 1 

959 1 

960 1 

961 1 

962 1 

963 1 

96i|. 1 

965 1 

966 1 

967 1 

968 1 

969 1 

970 1 

971 1 

972 1 

973 1 

974 1 



ERIC 



6455 


STRATFORD 


64 <56 


SECOITOARY 


64 <57 


COMPOSED 


6458 


QUALITY 


64.59 


COWUTATICN 


646 0 


COMPREG 


6461 


COMPOUND 


6462 


COMPOSITION 


6463 


COMPOSITE 


6464 


COUNTERROTATING 


6465 


COUNTERCURRENT 


6466 


CORRELATED 


6467 


CRITERION 


6468 


COUPON 


6469 


COUPLED 


6470 


SPECIMEN 


6471 


COROTAHMG 


6472 


STREA2-B 


6473 


COOLED 


64.74 


DIETZE 


6475 


DIATOMIC 


6476 


3TREAMTUBE 


6477 


DIAPHRAGM 


6478 


RAW 


6479 


DESTABILIZING 


6iE80 


DERIVATION 


6481 


SELECTED 


6482 


PROJECTION 


6483 


DEFICIENCY 


6484- 


STREET 


64-85 


PROTECTION 


6486 


DAYTII-iE 


6487 


DAYTONIGHT 


6488 


SPECIES 


6489 


DECREASE 


6490 


DECELERATING 


64-91 


DECAHNG 


6^v92 


SPANNING 


64^93 


PITCHUP 


6494 


DELAY 


6495 


REMOVED 


6496 


QUARTZ 


6497 


CYCLES 


6498 


POWELL 


6499 


DAMPOl-ETER 


6500 


SEEDED 


6501 


CROSSED 


6502 


CROSSECHON 


6503 


CROPPED 



371 
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TABLE* — continued ( 14 ) 
FOURTH QU/mTER DELETIONS 



Class 


Freq. 


Cum. Freq 


975 


1 


6504 


976 


1 


6505 


977 


1 


6506 


978 


1 


6507 


979 


1 


6508 


980 


1 


6509 


981 


1 


6510 


982 


1 


6511 


9? 3 


1 


6512 


98'i- 


1 


6513 


985 


1 


6514 . 


986 


1 


6515 


987 


1 


6516 


988 


1 


6517 


989 


1 


6518 


990 


1 


6519 


991 


1 


6520 


992 


1 


6521 


993 


1 


6522 


99'' 


1 


6523 


995 


1 


6524 


996 


1 


6525 


997 


1 


6526 


998 


1 


6527 


999 


1 


6528 


1000 


1 


6529 


1001 


1 


6530 


1002 


1 


6531 


1003 


1 


6532 


1004. 


1 


6533 


1005 


1 


6534 


1006 


1 


6535 


1007 


1 


6536 


1008 


1 


6537 


1009 


1 


6538 


1010 


1 


6539 


1011 


1 


6540 


1012 


1 


65/+1 


1013 


1 


6542 


101'.!. 


1 


6543 


1015 


1 


6544'. 


1016 


1 


65^+5 


1017 


1 


6546 


1018 


1 


65 / 4-7 


1019 


1 


65/48 


1020 


1 


6549 


1021 


1 


6550 


1022 


1 


6551 



ERIC 



372 



Term 

CURRENT 

CRUISING 

CROSSING 

CROSSFLW 

STRINGER 

DIRECTION 

DIFFUSION 

DIFFUSER 

DISEQUILIBRIUM 

DISCRETE 

DISCOVER 

DISCONTINUOUS 

DIRECT 

DIOHDE 

DEMENSIONLESS 

DIMEf^lON 

HORIZONTAL 

HOLE 

HOLDING 

HINGED 

SUBJECTED 

HEXACHLORETHANE 

HEl'lISPHERICALLY 

RELEASE 

HEDMOLZ 

HEAVY 

SELFINDUCED 

SELF 

SUBSTANTIAL 

IGNITED 

IDEALIZED 

SUDDEN 

HYDROCARBON 

HUiaDITY 

SEMELLIPSOID 

HUGONOIT 

HARTMANN 

HANDLING 

GYROSCOPIC 

GUN 

SUN 

SOUNDING 

PROMOTION 

GREEN 

PRACTICAL 

INCREASED 

INCLINED 

INCIDENT 



TABLEt — continued ( 15 ) 
FOURTH QUARTER DELETIONS 



Class 

1023 

1024 

1025 

1026 

1027 

1028 

1029 

1030 

1031 

1032 
1031 

103"- 
103 S 
1036 
1027 

1038 

1039 

lO^iO 
lO'-'-l 
lO'J-2 
10'’-3 
10''-5- 
10’!- S 
10’!-6 
1047 
10^.8 
10!-9 

1050 

1051 

1052 

1053 

1054 

1055 

1056 

1057 

1058 

1059 

1060 

1061 

1062 

1063 

1064 

1065 

1066 

1067 

1068 

1069 

1070 

1071 



ERIC 



Cum# Freq. 


Terra 


6552 


INBOARD 


6553 


IN 


6554 


Il'PROVED 


6555 


SUPERSATELLITE 


655 ^ 


INDICATOR 


6557 


INCREl-lENT 


6558 


INCREASING 


6559 


DUE 


6560 


DRY 


6561 


DROOP 


6562 


SUPPORTED 


6563 


DOmN 


6564 


DIVISION 


6565 


REUTIONSmP 


6566 


PERFORATED 


6567 


DRIERS 


6568 


IKIVEN 


6569 


DRIVE 


6570 


DRILLING 


6571 


DRILLED 


6572 


SOFT 


6573 


EIGEOTALVE 


657^+ 


ELECTRIC 


6575 


SUPPORTING 


6576 


ROOK 


6577 


ROOFTOP 


6578 


SENSOR 


6579 


PILOTING 


6580 


PHUGOID 


6581 


DISSIPATIVE 


6582 


DISPUCED 


6583 


YAV/ING 


6584 


WOOD 


6585 


SLUG 


6586 


RESTART 


6587 


DISTORTION 


6588 


Sl-iOOTH 


6589 


REST 


6590 


VJITHIN 


6591 


ELE^ 3 NTARY 


6592 


ELECTROI'AGNBTICAL 


6593 


ELECTROI^AGNETIC 


659 ^ 1 - 


END 


6595 


EI- 3 TTED 


6596 


SEI'IVERTEX 


6597 


PRODUCT 


6598 


EULINATICN 


6599 


ELEVON 


6600 


ELEVATED 



Freq 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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TABLE. — continued (l 6 ) 
FOURTH QUARTER DELETIOIB 



Class 


Freq. 


Cura. Freq. 


1072 


1 


6601 


107? 


1 


6602 


IO7A 


1 


6603 


1075 


1 


6604 


1076 


1 


660 s 


1077 


1 


6606 


1078 


1 


6607 


1079 


1 


6608 


1080 


1 


6609 


1081 


1 


6610 


1082 


1 


6611 


108 s 


1 


6612 


108i' 


1 


6613 


108 


1 


6614 


1086 


1 


6615 


1087 


1 


6616 


1088 


1 


6617 


1089 


1 


6618 


1090 


1 


6619 


1091 


1 


6620 


1092 


1 


6621 


1093 


1 


6622 


1094 


1 


6623 


109 


1 


6624 


1096 


1 


662 


1097 


1 


6626 


1098 


1 


6627 


1099 


1 


6628 


1100 


1 


6629 


1101 


1 


6630 


1102 


1 


6631 


1103 


1 


6632 


1104 


1 


6633 


1105 


1 


66311- 


1106 


1 


663^ 


1107 


1 


6636 


1108 


1 


6637 


1109 


1 


6638 


1110 


1 


6639 


1111 


1 


6640 


1112 


1 


6641 


1113 


1 


6642 


lllii- 


1 


6643 


111 


1 


66^14 


1116 


1 


6643 


1117 


1 


6646 


1118 


1 


6647 


1119 


1 


6648 
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Term 

WHIRL 

PHOTOI-IULTIPLIER 

FORGIlW 

WEESSINGER 

POLlfGON 

RESULTANT 

WEAKENING 

FREES TREAM 

FREEENTERIMG 

VfAVELENGTH 

SUPSTREAI^ 

FRICTIONLESS 

SETTING 

PROPAGATION 

FREON 

SUGHTLY 

POLAR 

FUSED 

PLASTIC 

GEAR 

GAUGE 

FUSION 

^■^ATER 

SYSTEi-iATTC 

SERVO 

GENERAL 

WARRE^a2 

RETARDING 

GRAPHiaL 

GRAIN 

WARD 

TABULATION 

GLASS 

GIVEN 

ROD 

GRAVITATIONAL 

GRATE 

PROPAGATING 

FLa/METER 

WAG!®R 

FLARE 

FLAPPING 

PORTION 

FLUTTERING 

VONKARI-yiN 

TAILBOOM 

FLEXURE 

FLFXURAL 



TABLE. --continued (l?) 
FOURTH QUARTER DELETIONS 



Class 


Freq. 


Cura. Preq. 


Term 


1120 


1 


6649 


VOLUl'IE 


1121 


1 


6610 


FIVES TAGE 


1122 


1 


6611 


FISSION 


1123 


1 


6612 


PRINCIPLE 


112iJ- 


1 


6653 


FILTER 


112 <5 


1 


66 l 4 


FILLING 


1126 


1 


6615 


VIBRATIONALLY 


1127 


1 


6616 


VIBRATIONAL 


1128 


1 


6617 


FIRING 


1129 


1 


6618 


FINAL 


1130 


1 


6619 


FEEDBACK 


nil 


1 


6660 


FAST 


1112 


1 


6661 


FAR 


1133 


1 


6662 


FALL 


111'''. 


1 


6663 


EXPRESSION 


1131 


1 


6664 


EXPONENTIAL 


1116 


1 


6661 


EXPONENT 


1137 


1 


6666 


VERY 


1138 


1 


6667 


SEVERELY 


1119 


1 


6668 


EXTERNALLY 


ll^-'O 


1 


6669 


EXTENT 


ini 


1 


6670 


EXTREME 


11'.'2 


1 


6671 


EXTRAPOLATED 


11^3 


1 


6672 


SIX 


llt'i- 


1 


6673 


FAIR 


11 '■’■1 


1 


667^ 


FAILURE 


H'16 


1 


6671 


FAHRENHEIT 


11'|.7 


1 


6676 


VENTURIS 


ll'(-8 


1 


6677 


ERROR 


H'19 


1 


6678 


EROSION 


1110 


1 


6679 


EQUIVALENCE 


nil 


1 


6680 


SKIP 


1112 


1 


6681 


ESTBATE 


1113 


1 


6682 


PROCEDURE 


lll^'l. 


1 


6683 


EQUIPMENT 


1115 


1 


6684 


EQUATORLAL 


1116 


1 


6681 


SKE>.''ED 


1117 


1 


6686 


SEVERE 


1118 


1 


6687 


ENVIRCNT-ENTAL 


1119 


1 


6688 


emtraix;e 


1160 


1 


6689 


EXOTHERl-OiC 


1161 


1 


6690 


EH TING 


1162 


1 


6691 


VARIATIONAL 


1163 


1 


6692 


TAILLESS 


1164 


1 


6693 


RSUY 


1165 


1 


6694 


EXCITED 


1166 


1 


6691 


TAILPLANE 


1167 


1 


6696 


POSITIVE 



375 



C 3 .c 15 S 

?-l68 

1169 

1170 

1171 

1172 

1171 

117’!- 

117*5 

1176 

1177 

1178 

1179 

1180 

1181 

1182 

1183 

1184 

1185 

1186 

1187 

1188 

1189 

1190 

1191 

1192 

119” 

1194 

119*5 

1196 

1197 

1198 

1199 

1200 

1201 

1202 

1202 

1204 

1205 

1206 

1207 

1208 

1209 

1210 

1211 

1212 

1213 

121 '.'. 

121 

1216 
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TABLE. — continued (I8) 
FOURTH QUARTER DELE.HONS 



Freq. 


Cum. Freq. 


TeT-m 


1 


6697 


EXACT 


1 


6698 


EVAPORATION 


1 


6699 


EVAPORATING 


1 


6700 


EVALUATION 


1 


6701 


VAPOURIZATION 


1 


6702 


I'SGH 


1 


6703 


iffinrui"- 


1 


6704 


VAPOUR 


1 


670 


>HDDLE 


1 


6706 


rlDCOURSE 


1 


6707 


MDAS 


1 


6708 


VANGUARD 


1 


6709 


PIPE 


1 


6710 


J'ECHAJ'OiSl': 


1 


6711 


’■ 3 THANE 


1 


6712 


l^iETSROID 


1 


6713 


I'STEORITE 


1 


6714 


VANSLSSS 


1 


6715 


POINTED 


1 


6716 


liOUI^TED 


1 


6717 


VALUED 


1 


6718 


MOLYBDENUI'i 


1 


6719 


I'iOL 


1 


6720 


VACUUI 4 


1 


6721 


TAP 


1 


6722 


TANGENT 


1 


6723 


UPRATING 


1 


6724 


mJNIND 


1 


672 <5 


UNTAPERED 


1 


6726 


TAPPING 


1 


6727 


PINE 


1 


6728 


TARGET 


1 


6729 


REDUCTED 


1 


6730 


REDUCED 


1 


6731 


FIS 3 I 0 N 


1 


6732 


MOISTURE 


1 


6733 


MODULUS 


1 


6734 


MODERATE 


1 


6735 


I-CCNOR 


1 


6736 


MNIMUM 


1 


6737 


I- 5 ILES 


1 


6738 


MILD 


1 


6739 


UN 5 IALUNG 


1 


6740 


NEUTRAL 


1 


6741 


NEARTRIANGULAR 


1 


6742 


UNSTALLED 


1 


6743 


UNITY 


1 


67 'i 4 


UNIT 


1 


6745 


NAUTICAL 



TABLE. — continued (19) 



Class 

1217 

1218 

1219 

1220 
1221 
1222 
1223 
1224 
122'= 
1226 
1227 
1228 

1229 

1230 

1231 

1232 

1233 

123''- 

1233 

1236 

’237 

1238 

1239 

12.'i0 

12'l1 

1242 

iz'n 

124'i. 

1243 

1246 

1247 

1248 

1249 

1250 

1231 

1232 

1233 
12?!- 

1235 

1236 

1237 

1238 

1239 

1260 

1261 

1262 

1263 

1264 

1265 



ERIC 



QUARTER DELETIONS 



n. Preq. 


Term 


6746 


TAYLOR 


6747 


SINGLY 


6748 


SHAPED 


673'-9 


NIGHTTIIffl 


6730 


SINGULAR 


6731 


NONCATALYTIC 


6732 


NONABLATING 


6733 


NOLIFT 


673i>- 


INSULATED 


6735 


INSTRlDiENTATION 


6736 


IN3TABIUTY 


6737 


UNIDTRECTIOKAL 


6738 


POSSION 


6739 


INTERIOR 


6760 


UNHEATED 


6761 


TELSi'-ETERING 


6762 


INTERNALLY 


6763 


INTERJECTORY 


6764 


REGRESSION 


6763 


INTEGRATING 


6766 


INTAKE 


6767 


UNFOLDING 


6768 


INJECTION 


6769 


INJECTED 


6770 


INITIATED 


6771 


lOTTIALLY 


6772 


INFINITBSII-llLLY 


6773 


RIGIDITY 


677'''- 


INFINITELY 


6773 


INEXORABLE 


6776 


ir®RTIA 


6777 


RIEI-IANN 


6778 


POSTBUCKLING 


6779 


INDIHDUAL 


6780 


INDIRECT 


6781 


UNDISSOCIATED 


6782 


umderi-:ater 


6783 


INTERPU^ETARY 


6784 


UNCOUPLED 


6783 


ISOBAR 


6786 


IRROTATIONAL 


6787 


RIGIDLY 


6788 


ION 


6789 


IODIDE 


6790 


ISOTROPIC 


6791 


ISOTOPE 


6792 


ISOENERGETIC 


6793 


ISOBARIC 


6794 


UNCAI-IBERED 



FOURTH 

Fl*eqi 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 ' 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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TABLE# — continued (20) 



Class 

1266 

126? 

1268 

1269 

1270 

1271 

1272 

1273 

1274 

1275 

1276 
" 1277 

1278 

1279 

1280 
1281 
1282 
1283 
128 '.! 

128 <5 

1286 

1287 

1288 

1289 

1290 

1291 

1292 

1293 

129 -''- 

129 

1296 

1297 

1298 

1299 

1300 

1301 

1302 

1303 

130^^ 

130 7 

1306 

1307 

1308 

1309 

1310 

1311 

1312 

1513 

131't 

Er|c 



QUARTER DELETIONS 



1 . Freq. 


Term 


6795 


UNBOUNDED 


6796 


ULTRAVIOLET 


6797 


TAfODIMENSIONS 


6798 


KUSSNER 


6799 


HNK 


6800 


THEODORSEN 


6801 


JEFEERYHAMEL 


6802 


ISSUING 


6803 


KARI-IAN 


6804 


JUI'IO 


6805 


JouKa«;sn 


6806 


JOHAMNESEN 


6807 


ORDINATE 


6808 


TURNING 


6809 


opni'iui/: 


6810 


OPTIMIZATION 


6811 


THEOREM 


6812 


REAR'-rARD 


6813 


RANKLINE 


681/i- 


OSCILLOGRAPH 


681^5 


ORTHOGONAL 


6816 


ORIGINATING 


6817 


ORIFICE 


6818 


ORIENTATION 


6819 


TURBULENCE 


6820 


TURBOMCHB3E 


6821 


Or®DIMENSlON 


6822 


SIt'iPLY 


6823 


DILFLa,-J 


6824 


OCTAGONAL 


6825 


TUNGSTEN 


6826 


OBUTENSSS 


6827 


SIMPLE 


6828 


NUCLEAR 


6829 


NPL 


6830 


OVERRELAXATION 


6831 


OVEREXPANDED 


6832 


OVERALL 


6833 


TRUNCATED 


6834. 


SIHLITUDE 


683*5 


PAIR 


6836 


TRIANGLE 


6837 


THICKENED 


6838 


OUTFLa^* 


6839 


THERMALLY 


68^1-0 


SIMILARITY 


6841 


TRAVERSE 


6842 


SHIELDING 


6843 


REFERENCE 



FOURTH 

Freq 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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TABLE. — continued (21) 
FOURTH QUARTER DELETIONS 



Class 


Freq. 


Cum. Freq. 


Term 


131 <5 


1 


68^d.i- 


TRAVELLING 


1316 


1 


6845 


NONPERFECT 


1317 


1 


6846 


KONPARALLEL 


1318 


1 


6847 


TRANSVERSE 


1319 


1 


6848 


.NOTSOSLENIER 


1330 


1 


684-9 


NOSEPIECE 


1321 


1 


6850 


SILICA 


1322 


1 


^851 


SHORT 


1323 


1 


6852 


NONDISSIPATIVE 


1324 


1 


6853 


NONDISSIPATING 


132.'^ 


1 


685^j- 


TRANSPIRATION 


1326 


1 


685.3 


2 -lAGNESIUI-.' 


1327 


1 


6856 


LUNISOLAR 


1328 


1 


6837 


SIDBNASH 


1329 


1 


6858 


LORENZ 


1330 


1 


6859 


THREE 


1331 


1 


6860 


LUI'INOUS 


13^2 


1 


6861 


LUllENOSITY 


1331 


1 


6662, 


P 0 I 3 EDILLE 


13 3 ^' 


1 


6863 


LOCKHEED 


133 «^ 


1 


6864 


LOCALLY 


1336 


1 


6663 


KARTIAN 


1337 


1 


6666 


2 IA? 20 nETER 


1338 


1 


6867 


TRANSF 0 R 1 -ATICI-. 


13^9 


1 


6868 


I'iAJOR 


iV'O 


1 


6869 


i'AGNITUDS 


13 ’!l 


1 


6870 


RAI^P 


13^2 


1 


6871 


MTRIX 


13-''-3 


i 


6872 


NAGNIFICATION 


13'i2l 


1 


6873 


MAGNETOPUSi-A 


13 '!-'; 


1 


667/-:- 


MAGNETOFLUIDI'ECH 


13 ^ 1-6 


1 


6875 


I'iAGNETIC 


13'’7 


1 


6876 


TRAJECTORIES 


13.''.8 


1 


6677 


PROPELLED 


13'!-9 


1 


6878 


LOADED 


13‘?0 


1 


6879 


TORSO 


1351 


1 


6880 


REVERSED 


1332 


1 


6881 


LIFE 


1333 


1 


6882 


REVERSIBILITY 


133 il. 


1 


6883 


LIQUIFACTICW 


1335 


1 


6884 


LINERS 


l'=^6 


1 


6883 


LINEARIZATION 


1337 


1 


6886 


TORQUE 


1338 


1 


6887 


THdAITES 


1339 


1 


6888 


REENTRANT 


1360 


1 


6889 


LEAD 


1361 


1 


6890 


TOLU-ENSCHLICHT 


1362 


1 


6891 


LIBRATION 



er|c 
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TABLE. — continued ( 22 ) 
FOURTH QUARTER DEXETIOJB 



Class 


Freq. 


Cura. Freq. 


Term 


1363 


1 


6892 


LESS 


1364 


1 


6893 


TOLL'-ilEN 


136*5 


1 


6894 


Lk\! 


1366 


1 


6895 


UTERALLY 


1367 


1 


6896 


TIHEOPTIMITM 


1368 


1 


6897 


PUNET 


1369 


1 


6898 


LAI'INARIZATION 


1370 


1 


6899 


L-IETHOD 



ERIC 380 
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375 



ablation 


1978 


2099 


2100 


2101 












about 


1792 


















action 




















accuracy 




















addition 


1974 


















aerodynamic 


1360 


1406 


1415 


1594 


1597 


1598 


1606 


1608 


1613 




1615 


1671 


1675 


1677 


1687 


1688 


1698 


1704 


1707 




1708 


1709 


1711 


1712 


1717 


1748 


1783 


1792 


1798 




1919 


1972 


1981 


1982 


1995 


2104 


2339 


2342 


2379 




2391 


















aerofoil 


1316 


1409 


1416 


1467 


1597 


1672 


1681 


1687 


1701 




1702 


1703 


1706 


1797 


1798 


1799 


2153 






aeroplane 


1783 


















air 


1302 


1416 


1437 


1576 


1578 


1588 


1592 


1613 


1614 




1615 


1616 


1617 


1619 


1620 


1621 


1622 


1672 


1691 




1695 


1772 


1874 


1970 


1991 


1992 


1995 


1997 


2083 




2101 


2157 


2274 


2313 


2379 










alone 




















altitude 


1302 


1574 


1578 


1606 


1620 


1621 


1717 


1719 


1971 




1983 


2102 


2103 


2150 


2274 


2379 


2391 






analysis 


1360 


1437 


1576 


1588 


1589 


1590 


1596 


1666 


1680 




1836 


1974 


1984 


1988 


1989 


2099 


2100 


2317 


2339 




2391 


















analytical 


1606 


1613 


1670 


1719 


2187 











er|c 



382 



376 



angle 


1360 


1420 


1443 


1572 1575 


1590 


1592 


1619 


1655 




1672 


1681 


1688 


1693 1694 


1695 


1698 


1704 


1707 




1708 


1709 


1711 


1712 1713 


1717 


1719 


1772 


1782 




1786 


1788 


1792 


1973 1985 


1988 


1989 1993 


1997 




2001 


2074 


2075 


2077 2154 


2274 


2339 2341 


2364 




2379 
















apex 


1694 
















area 


1590 


1788 


1799 












assumed 


















assumption 

atmosph:ire 


1302 


1436 


1509 


1613 1614 


1615 


1616 


1617 


1619 




1620 


1621 


1622 


1717 1719 


1982 


1983 


2001 


2077 




2102 


2103 


2391 












axial 


1588 


1589 


1590 


1592 1709 


1772 


1836 1967 


1987 




1988 


1995 


2001 












blast 


















Blasius 


1320 


1321 


1322 


1476 










boattail 


1360 


1713 


1997 












body 


1317 


1360 


1572 


1574 15 75 


1576 


1578 1606 


1655 




1666 


1670 


1672 


1681 1688 


1710 


1717 


1719 


1792 




1921 


1966 


1967 


1973 19 78 


1981 


1983 


1991 


1992 




1993 


1994 


1995 


2002 2074 


2075 


2099 


2104 


2111 




2153 


2155 


2157 


2274 2318 


2319 


2321 


2391 





383 



377 



boundary 


1302 


1311 


1316 


1320 


1321 


1322 


1324 


1335 




1378 


1383 


1406 


1415 


1416 


1436 


1437 


1476 




1569 


1572 


1576 


1606 


1608 


1655 


1666 


1667 




1671 


1672 


1675 


1696 


1710 


1728 


1785 


1787 




1788 


1792 


1793 


1794 


1796 


1797 


1798 


1799 




1879 


1964 


1965 


1966 


1972 


1973 


1974 


1978 




1981 


1982 


1992 


1997 


2076 


2080 


2081 


2082 




2083 


2087 


2088 


2099 


2100 


2154 


2155 


2157 




2187 


2274 


2313 


2317 


2319 


2321 


2322 


2364 




2367 
















calculated 


















calculating 


1677 
















channel 


1351 


1963 


1966 


2083 


2084 








Chapman 


















change 


1590 


1616 


1617 


1622 


1699 


1968 


2101 


2154 


choking 


1590 


1672 


1799 


2154 


2155 








close 


















closely 


















coefficient 


1302 


1311 


1316 


1360 


1383 


1415 


1443 


1451 




1569 


1588 


1590 


1592 


1655 


1671 


1672 


1687 




1688 


1693 


1694 


1696 


1699 


1700 


1701 


1702 




1704 


1707 


1709 


1711 


1713 


1719 


1772 


1779 




1794 


1797 


1798 


1836 


1919 


1964 


1965 


1980 




1992 


1995 


2078 


2080 


2081 


2082 


2084 


2321 




2342 


2391 















er|c 



384 



378 



combustion 


1691 


1978 


2100 












comparison 


















compressible 


1383 


1406 


1409 


1676 


1680 


1701 


1703 


1705 




1748 


1779 


1967 


1984 


1985 


1990 


2076 


2100 


compressor 


1588 


1589 


1590 


1591 


1592 


1772 


1984 


1985 




1986 


1987 


1990 












computer 


1321 


1677 
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Cutoff is th'^ number of terms a document must have in common with a search question to be considered retrieved 
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Cutoff is the number of terms a document must have in common with a search question to be considered retrieved 
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The difference is significant at .05. Cutoff is the number of terms a document must have in 

common with a search question to be considered retrieved 

The difference is significant at .01. 
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* The difference is significant at .05* 

** The difference is significant at .01. 

X Sample size too small, owing to indeterminate changes in precis ion, to warrant any conclusion about significance. 
Cutoff is the number of terms a document must have in common with a search question to be considered retrieved. 
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